{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归到神经网络\n",
    "\n",
    "* 线性回归是最简单的神经网络形式。可以把神经网络看成是线性回归的一种扩展。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"14.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(url= \"14.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 为什么神经网络前先讲线性回归？迭代法解决线性回归问题，可以理解梯度下降法。\n",
    "\n",
    "* 迭代法：假设，判断，修正，再判断，再修正，直到满足条件。\n",
    "\n",
    "* 为什么损失函数用平方差；\n",
    "\n",
    "* 多种迭代法；梯度下降的理解，为什么减去梯度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归的神经网络理解\n",
    "\n",
    "$$y = b + \\omega_1 x_1 + \\omega_2 x_2 + ... + \\omega_p x_p$$\n",
    "\n",
    "$$y = b + \\omega_1 x_1 + \\omega_2 x_2 + ... + \\omega_p x_p + \\epsilon$$\n",
    "\n",
    "其中:\n",
    "\n",
    "$y$ 是因变量,需要预测的目标\n",
    "$x_1, x_2, ..., x_p$ 是自变量,输入特征\n",
    "$b$ 是截距项\n",
    "$\\omega_1, \\omega_2, ..., \\omega_p$ 是回归系数\n",
    "$\\epsilon$ 是误差项\n",
    "\n",
    "模型的目标是找到一组参数,使得预测$\\hat{z}$尽可能接近真实观测值z。\n",
    "常见的训练方法是最小二乘法,通过最小化所有样本的平方误差来求解最优参数。\n",
    "\n",
    "多元线性回归广泛用于数值预测任务,可以看作是最简单的机器学习模型之一。但表达能力有限,只能表示线性关系。\n",
    "\n",
    "\n",
    "### 线性回归的拟合过程（神经网络视角）\n",
    "\n",
    "https://www.mladdict.com/\n",
    "\n",
    "* 假设系统可以通过线性关系来描述\n",
    "\n",
    "* 先随机设定线性方程的参数，并算出对应结果\n",
    "\n",
    "* 计算误差（均方差损失），并调整参数（牛顿切线法）\n",
    "\n",
    "最小二乘法误差计算公式：\n",
    "$$\n",
    "\\text{MSE} = \\frac{1}{n} \\sum_{i=1}^{n} (y_i - \\hat{y}_i)^2\n",
    "$$\n",
    "\n",
    "其中，$y_i$ 是第 $i$ 个样本的真实值，$\\hat{y}_i$ 是第 $i$ 个样本的预测值，$n$ 是样本数量。\n",
    "\n",
    "均方差损失（Mean Squared Error, MSE）是回归问题中常用的损失函数，用于衡量预测值与真实值之间的差异。MSE 的值越小，表示模型的预测效果越好。\n",
    "\n",
    "上述最小二乘法误差公式中y需要用x和参数计算得到，即y=bx+e，其中e是误差项。如此公式变为\n",
    "\n",
    "$$\n",
    "\\text{MSE} = \\frac{1}{n} \\sum_{i=1}^{n} (y_i - (b + \\omega_1 x_{i1} + \\omega_2 x_{i2} + ... + \\omega_p x_{ip}))^2\n",
    "$$\n",
    "\n",
    "如果x只有一个维度，即x_i，则公式变为\n",
    "\n",
    "$$\n",
    "\\text{MSE} = \\frac{1}{n} \\sum_{i=1}^{n} (y_i - (b + \\omega x_{i}))^2\n",
    "$$\n",
    "\n",
    "其中，$x_{i1}$ 是第 $i$ 个样本的第 1 个特征值。\n",
    "\n",
    "上述公式对$\\omega$求导结果为\n",
    "\n",
    "当我们使用最小二乘法进行线性回归时，假设模型形式为 \\( y = wx + b \\)，其中 \\( w \\) 是权重（或斜率），\\( b \\) 是偏置（或截距），而 \\( x \\) 和 \\( y \\) 分别是输入和输出变量。最小二乘法的目标是最小化预测值与实际值之间的平方误差之和。\n",
    "\n",
    "损失函数 \\( J(w, b) \\) 通常定义为所有数据点的残差平方和的一半，即：\n",
    "$$ J(w, b) = \\frac{1}{2n} \\sum_{i=1}^{n} (y_i - (wx_i + b))^2 $$\n",
    "\n",
    "\n",
    "首先，我们对 \\( w \\) 求偏导：\n",
    "$$ \\frac{\\partial J}{\\partial w} = \\frac{1}{2n} \\sum_{i=1}^{n} -2x_i(y_i - (wx_i + b)) = \\frac{1}{n} \\sum_{i=1}^{n} -x_i(y_i - wx_i - b) $$\n",
    "\n",
    "然后，我们对 \\( b \\) 求偏导：\n",
    "$$ \\frac{\\partial J}{\\partial b} = \\frac{1}{2n} \\sum_{i=1}^{n} -2(y_i - (wx_i + b)) = \\frac{1}{n} \\sum_{i=1}^{n} -(y_i - wx_i - b) $$\n",
    "\n",
    "简化后得到：\n",
    "$$ \\frac{\\partial J}{\\partial w} = \\frac{1}{n} \\sum_{i=1}^{n} x_i(wx_i + b - y_i) $$\n",
    "$$ \\frac{\\partial J}{\\partial b} = \\frac{1}{n} \\sum_{i=1}^{n} (wx_i + b - y_i) $$\n",
    "\n",
    "为了找到最小值点，我们将这两个偏导数分别设为零，并解方程组来找到 \\( w \\) 和 \\( b \\) 的值。在实际应用中，这通常通过迭代优化算法（如梯度下降）来实现。\n",
    "\n",
    "\n",
    "* 重复上述过程，直到误差达到最小\n",
    "\n",
    "https://www.zhihu.com/question/305638940\n",
    "\n",
    "https://baike.baidu.com/item/%E5%88%87%E7%BA%BF%E6%B3%95/2428699?fr=ge_ala\n",
    "\n",
    "\n",
    "### 核心概念\n",
    "\n",
    "* 误差——均方差\n",
    "\n",
    "* 导数\n",
    "\n",
    "* 向前传播\n",
    "\n",
    "* 反向传播\n",
    "\n",
    "* 梯度下降\n",
    "\n",
    "* 损失函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 梯度下降\n",
    "\n",
    "常见的迭代法\n",
    "\n",
    "https://www.cnblogs.com/shixiangwan/p/7532830.html\n",
    "\n",
    "\n",
    "梯度下降\n",
    "\n",
    "https://www.zhihu.com/question/305638940\n",
    "\n",
    "\n",
    "* 梯度向量\n",
    "\n",
    "梯度向量的方向和大小确定方式可以用以下公式解释:\n",
    "\n",
    "假设有目标函数$f(\\mathbf{x})$,其中$\\mathbf{x}$是$n$维向量。\n",
    "\n",
    "则目标函数$f$相对于$\\mathbf{x}$的梯度是一个向量:\n",
    "\n",
    "$$\\nabla_{\\mathbf{x}} f = \\begin{bmatrix} \\frac{\\partial f}{\\partial x_1} \\\\ \\frac{\\partial f}{\\partial x_2} \\\\ \\vdots \\\\ \\frac{\\partial f}{\\partial x_n} \\end{bmatrix}$$\n",
    "\n",
    "其中:\n",
    "\n",
    "- $\\frac{\\partial f}{\\partial x_i}$表示$f$相对于$x_i$的偏导数\n",
    "\n",
    "- 梯度向量的每个元素表示目标函数相对于对应变量的变化率\n",
    "\n",
    "梯度向量方向性质:\n",
    "\n",
    "- 梯度方向与目标函数增长最快的方向相反\n",
    "\n",
    "- 沿负梯度方向移动可以使目标函数值下降最快\n",
    "\n",
    "梯度向量大小性质:\n",
    "\n",
    "- 梯度大小表示目标函数沿该方向的变化率\n",
    "\n",
    "- 梯度越大,目标函数越“陡峭”\n",
    "\n",
    "所以梯度向量充分表示了目标函数的局部变化,方向和大小的合理利用都有利于设计优化算法。\n",
    "\n",
    "\n",
    "\n",
    "### 对于权重参数的梯度下降的更新公式\n",
    "\n",
    "假设一个优化问题,其目标是最小化损失函数$L(\\omega)$,其中ω表示模型的参数。\n",
    "\n",
    "则梯度下降法的迭代更新公式为:\n",
    "\n",
    "$$\\omega^{(t+1)} = \\omega^{(t)} - \\eta \\nabla_{\\omega} L(\\omega^{(t)})$$\n",
    "\n",
    "其中:\n",
    "\n",
    "- $\\omega^{(t)}$: 模型参数在第$t$次迭代的值  \n",
    "- $\\eta$: 学习率,确定更新步长\n",
    "- $\\nabla_{\\omega} L(\\omega^{(t)})$: $L$相对于$\\omega$的梯度向量\n",
    "- $t$: 迭代次数\n",
    "\n",
    "梯度下降的迭代过程是:\n",
    "\n",
    "1. 初始化参数$\\omega$ \n",
    "\n",
    "2. 计算损失函数$L(\\omega)$关于参数$\\omega$的梯度$\\nabla_{\\omega} L(\\omega)$\n",
    "\n",
    "3. 沿负梯度方向更新参数,步长为学习率$\\eta$\n",
    "\n",
    "4. 重复2-3,直到损失函数收敛或达到预定迭代次数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"20.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(url= \"20.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 课堂作业\n",
    "\n",
    "假设有个样本集，其中包含一个特征和一个目标值。请使用线性回归模型来拟合这个样本集，假设一开始参数为（1，1） 把参数带入某个样本计算损失函数的值，并计算导数，然后更新参数，然后求新的损失函数值\n",
    "\n",
    "样本集如下：\n",
    "\n",
    "| 特征1 |目标值 |\n",
    "| --- |--- |\n",
    "| 1.0 | 3.0 |\n",
    "| 3.0 | 5.0 |\n",
    "| 5.0 | 7.0 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "88.5"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(95+82)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y=b+wx\n",
    "b=0.5\n",
    "w=0.5\n",
    "0.5+0.5*x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "0.5+0.5*1.0\n",
    "0.5+0.5*3.0\n",
    "0.5+0.5*5.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.3\n",
      "5.3\n",
      "8.3\n"
     ]
    }
   ],
   "source": [
    "print(1.5*1.0+0.8)\n",
    "print(1.5*3.0+0.8)\n",
    "print(1.5*5.0+0.8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7566666666666674"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "((3-2.3)**2+(5-5.3)**2+(7-8.3)**2)/3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 从两层到多层——深度学习的起源\n",
    "\n",
    "* 主要解决了参数量的问题，越是复杂的系统所需要的参数量必然是很大的，而线性回归的参数量是固定的，所以需要多层来增加参数量\n",
    "\n",
    "* 多层网络需要注意输入输出的维度衔接问题\n",
    "\n",
    "### 线性回归的图形表达\n",
    "\n",
    "* 输入点代表特征值\n",
    "\n",
    "* 一条边代表一个$x*w$\n",
    "\n",
    "* 输出点代表把每条边的结果相加后的值\n",
    "\n",
    "https://nnplayground.com/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"191.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"191.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 神经网络前向传播的代码实现\n",
    "\n",
    "* 简单代码实现\n",
    "\n",
    "torch.tensor\n",
    "\n",
    "torch.mv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"output1.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"output1.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2.],\n",
       "        [3., 4.],\n",
       "        [5., 6.]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]],dtype = torch.float32)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.],\n",
       "        [2.],\n",
       "        [3.]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y=torch.tensor([[1.0],[2.0],[3.0]],dtype = torch.float32)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "w=torch.tensor([-0.0588,  0.6530],dtype = torch.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.2472, 2.4356, 3.6240])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.mv(x,w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 通过torch.nn.Linear()函数构建隐藏层\n",
    "\n",
    "torch.nn.Linear\n",
    "\n",
    "output.weight\n",
    "\n",
    "torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]],dtype = torch.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]],dtype = torch.float32)\n",
    "output0=torch.nn.Linear(2,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[-0.1620,  0.0175]], requires_grad=True)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output0.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.4798],\n",
       "        [-0.7686],\n",
       "        [-1.0574]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output0(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.1624, -0.1619]], requires_grad=True)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output0.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(20., grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([[2.,4.]], requires_grad=True)\n",
    "y = x**2\n",
    "z=y.sum()\n",
    "z.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[4., 8.]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<PowBackward0 object at 0x00000207C161BCD0>\n",
      "tensor(4.)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# 创建一个张量\n",
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "\n",
    "# 进行一些操作\n",
    "y = x**2\n",
    "\n",
    "print(y.grad_fn)  # 输出: <torch.autograd.function.PowBackward0 object at 0x7f8e3c0a8f70>\n",
    "\n",
    "y.backward()\n",
    "\n",
    "print(x.grad)  # 输出: 4.0\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.1624, -0.1619]], requires_grad=True)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output0.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.6937],\n",
       "        [1.2833],\n",
       "        [1.8729]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]],dtype = torch.float32)\n",
    "output0=torch.nn.Linear(2,1)\n",
    "output0(x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.3967, -0.1019]], requires_grad=True)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output0.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多层神经网络的传播实现\n",
    "\n",
    "* 神经元与隐藏层的增减的效果（chatgpt的回答）\n",
    "\n",
    "在构建深度学习模型时,增加单层神经元数量和增加隐藏层数量这两种方法都可以增强网络的表达能力,但带来的效果有些差异:\n",
    "\n",
    "1. 增加单层神经元数量,主要作用是提升该层对更多特征的学习表达能力。适合当前层特征不够情况。\n",
    "\n",
    "2. 增加隐藏层数量,主要作用是堆叠更多的非线性变换,提升全网络的层次化特征学习能力。适合功能表达不足情况。\n",
    "\n",
    "3. 增加单层神经元数量对网络宽度产生影响,而增加隐藏层对网络深度产生影响。\n",
    "\n",
    "4. 单层神经元增多容易导致过拟合;而层数增多可以通过有效的预训练及正则化方法减轻过拟合。\n",
    "\n",
    "5. 层数增多会使得训练时间加长,而单层神经元增加影响较小。\n",
    "\n",
    "综上,增多神经元数量的效果局限在单层,而增加层数影响全网;当特征表示力不足时,可以适当加大单层宽度;当层次表示不够时,可以适当加深网络。需要根据具体情况选择最优的网络结构。\n",
    "\n",
    "* 各层之间的衔接以及输出层\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.1974],\n",
       "        [0.2597],\n",
       "        [0.3221]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([[1.0,2.0],[3.0,4.0],[5.0,6.0]],dtype = torch.float32)\n",
    "h1=torch.nn.Linear(2,3)\n",
    "h2=torch.nn.Linear(3,4)\n",
    "h3=torch.nn.Linear(4,1)\n",
    "output=h1(x)\n",
    "output=h2(output)\n",
    "output=h3(output)\n",
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"191.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"191.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 无论是增加神经元或是隐藏层，都无法改变模型线性的本质\n",
    "\n",
    "* 线性模型无法拟合非线性关系\n",
    "\n",
    "* 隐藏层激活函数的引入使得模型能够拟合非线性关系（回归问题只需要隐藏层引入激活函数）\n",
    "\n",
    "* relu的特性\n",
    "\n",
    "* relu的pytorch实现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 从线性到非线性——激活函数的引入\n",
    "\n",
    "### 隐藏层激活函数\n",
    "\n",
    "* 隐藏层激活函数的作用是引入非线性，使得模型能够拟合非线性关系\n",
    "\n",
    "线性回归:\n",
    "\n",
    "$$ y = \\beta X + b $$\n",
    "\n",
    "其中,$X$是输入特征,$y$是输出,$b$是偏置项,$\\beta$是线性回归的参数。线性回归假设输入和输出之间是线性关系。\n",
    "\n",
    "单层神经网络:\n",
    "\n",
    "$$ y = f(X\\beta + b) $$\n",
    "\n",
    "其中$f$是一个非线性激活函数,如sigmoid。单层网络在线性回归基础上加了非线性变换。\n",
    "\n",
    "多层神经网络:\n",
    "\n",
    "$$ y = f_n(W_n...f_2(W_2f_1(W_1X+b_1)...)+b_n) $$\n",
    "\n",
    "其中$W_i, b_i$表示每一层的权重和偏置,$f_i$表示激活函数。多层网络堆叠线性变换和非线性变换,能够拟合更复杂的函数。\n",
    "\n",
    "从上式可以看出,线性回归是最简单的单层神经网络。随着层数的增加和非线性的引入,神经网络可以表示更加复杂的函数,拟合能力也越强。两者之间是渐进的关系。\n",
    "\n",
    "* 常用的隐藏层激活函数\n",
    "\n",
    "ReLU（Rectified Linear Unit）: $$ f(x) = \\max(0, x) $$\n",
    "\n",
    "* 隐藏层的激活函数主要有以下几个作用（chatgpt）:\n",
    "\n",
    "1. 引入非线性:\n",
    "激活函数的引入让网络可以拟合复杂的非线性函数,而不仅是简单的线性映射,这大大提高了模型的表达能力。\n",
    "\n",
    "2. 定义输出范围:\n",
    "不同的激活函数将输出值映射到不同的范围,如Sigmoid将实数映射到[0,1],tanh映射到[-1,1]。这为后续层提供合适数值范围的输入。\n",
    "\n",
    "3. 实现层间转换:\n",
    "激活函数实现了层与层之间的转换,一个层的输出通过激活函数成为下层的输入,完成信息的传递和转换。\n",
    "\n",
    "4. 提供可训练参数:\n",
    "激活函数的参数也可以加入训练,如批归一化层中的可学习参数,提高模型的表达能力。\n",
    "\n",
    "5. 提高泛化能力:\n",
    "激活函数增强了模型对未见样本的泛化能力,如ReLU具有稀疏化作用,减少了过拟合风险。\n",
    "\n",
    "6. 提供梯度:\n",
    "激活函数可导,为网络的反向传播提供了梯度,使网络可以通过梯度下降进行训练。\n",
    "\n",
    "7. 提供正则化:\n",
    "某些激活函数如ReLU也具有一定的正则化作用。\n",
    "\n",
    "所以隐藏层的激活函数对网络性能有重要影响,它实现了层与层之间的非线性变换,是网络可以拟合复杂函数的关键。\n",
    "\n",
    "* 机器学习中常用的激活函数\n",
    "\n",
    "机器学习中常用的激活函数只有恒等函数（identity function），阶跃函数（sign），sigmoid\n",
    "函数，ReLU，tanh，softmax这六种，其中Softmax与恒等函数几乎不会出现在隐藏层上，Sign、Tanh\n",
    "几乎不会出现在输出层上，ReLU与Sigmoid则是两种层都会出现，并且应用广泛。在这里，我们将总结性声明一下输出层的g(z)与隐藏层\n",
    "的h(z)之间的区别，以帮助大家获得更深的理解：\n",
    "\n",
    "1. 虽然都是激活函数，但隐藏层和输出层上的激活函数作用是完全不一样的。输出层的激活函数\n",
    "是为了让神经网络能够输出不同类型的标签而存在的。其中恒等函数用于回归，sigmoid函数用于\n",
    "二分类，softmax用于多分类。换句说， 仅仅与输出结果的表现形式有关，与神经网络的效果\n",
    "无关，也因此它可以使用线性的恒等函数。但隐藏层的激活函数就不同了，如我们之前尝试的\n",
    "XOR，隐藏层上的激活函数 的选择会影响神经网络的效果，而线性的 是会让神经网络的结\n",
    "构失效的。\n",
    "\n",
    "2. 在同一个神经网络中，输出层与隐藏层的激活函数可以是不同的，并且在大多数运行回归和多分类的神经网络时，他们也的确是不同的。每层上的 可以是不同的，但是同一层上的激活函数必须一致。\n",
    "\n",
    "\n",
    "* 前向传播的标准实现形式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=torch.tensor(range(-10,10),dtype = torch.float32)\n",
    "y=torch.relu(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1a942bbc340>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA08AAAFfCAYAAACWd/eoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABRjUlEQVR4nO3deXyU9bn///dkJUASQkI2CElkCyQIISgkCogoQVHqUhGxGk6VShV3v1pPTw+kv1rQ42k9p6eKCwdZVLBFWzzYBlDAKpEiBE0IRJA9CxEMGdYsM/fvjztkIdsEMkuS1/PxmAfe13w+k2uGm5grn/u+PhbDMAwBAAAAAFrk5e4EAAAAAKAjoHgCAAAAAAdQPAEAAACAAyieAAAAAMABFE8AAAAA4ACKJwAAAABwAMUTAAAAADjAx90JuIPdbldRUZECAwNlsVjcnQ4AAAAANzEMQ6dOnVJ0dLS8vFpeW+qSxVNRUZFiYmLcnQYAAAAAD3HkyBH169evxTFdsngKDAyUZH5AQUFBbs4GAAAAgLtYrVbFxMTU1ggt6ZLF04VL9YKCgiieAAAAADh0Ow8NIwAAAADAARRPAAAAAOAAiicAAAAAcADFEwAAAAA4gOIJAAAAABxA8QQAAAAADuiSrcoBAAAAuIetulp7tmbpXFmhAkL6KmFMurx9OkZZ0jGyBAAAANDh5WQtVXR2phJ1ojZ2bH2oilLnKTk9w42ZOYbiCQAAAIDT5WQt1Ygtj5kH9faj7WOcUJ8tjylH8vgCinueAAAAADiVrbpa0dmZkiQvS8PnLhxHZWfKVl3t4szahuIJAAAAgFPt2ZqlCJ1oVDhd4GWRInVCe7ZmuTaxNqJ4AgAAAOBUp0v2OjTuXFmhkzO5PNzzBAAAAKDdGXa7vt2xSeWfv6kRZesb3OfUnICQvs5P7DJQPAEAAABoN+Vlx7Un6031+XalhtgPmkGLVGV4yUd2WZooouyGVGoJVcKYdJfm2lYUTwAAAAAui2G3q+CrT3Tqi7eUdPJTjbFUSpLOG77K7XW9AtMe1NkfijTyy8dlGA2bRtgN88/i1HmK9PD9njw7OwAAAAAeq/zEMe3OelOR+1YqwX7EDFqkA16xOjZohoamz9ZVvfvUjs/xsig6O1MR9fZ5KrWEqriD7PNkMQzDcHcSrma1WhUcHKzy8nIFBQW5Ox0AAACgwzDsdu3emqWz2W9pePlm+VuqJEnnDD/lhtygoGsf1JBRE2Xxaro3na26Wnu2ZulcWaECQvoqYUy6vN244tSW2oCVJwAAAACtKvu+WAXr3lD0d6s0zF7TFc8ifecdr+NDZmro5Ad0da/QVl/H28dHiddMdXK2zkHxBAAAAKBJht2uXdlrVfHl/2q49TONtZib2J41/JXX+0b1GvczDRo5TgOaWWXqbCieAAAAADRw4thR7V33hvru/5OSjCIzaJH2eg/UDwkzlZj+U10dFOLeJN2A4gkAAACA7Dabdn2xRlX/XKKkU59rrMUmSTptBGhXWLpCx8/WoBHXujlL96J4AgAAADopR5ozHC85rL1ZixR78M8abhwzgxbpW5/BOjn0XiVOnqUxgb1cn7wHongCAAAAOqGcrKWKzs5UYr224MfWh6oodZ6unHSvdv3jL7J9tURJp7OVWrPKdMoIUH6fmxQ24SENHj7WXal7LFqV06ocAAAAnUxO1lKN2PKYpMYb0loklVmC1FvW2vgen6GyJt6rpBvvV/eewS7O1r1oVQ4AAAB0UbbqakVnZ0pqWDjVP+4tq6xGd+WHT1XExIeUMOwqF2fZMVE8AQAAAJ3Inq1Z5qV6lpbHHZz4Pxp73Z2uSaqT6BoN2QEAAIAuoLqqUtZvPnJobOXpE60PQgOsPAEAAAAdXNHBAh3asEgDjv5FqfrBoTkBIX2dnFXnQ/EEAAAAdEBVlRXK3fi+fHKWKuncV4q2mH3gytRTvka1uut8o3ueJLNpRKklVAlj0l2cccdH8QQAAAB0IIX7d+vwhtc0qOivGqWTZtAi5fmPVMWI+5V0/T3K3/wnjdjymOxG4257klScOk+RPpQCbeX2e57i4uJksVgaPR555JEmx2/atKnJ8Xv27HFx5gAAAIBrVFac1/aPlyh3wUT1XTZWqUVLFaaTOqFgZUffr8L7tyjp+c1KufkB+XfrruT0DH2d9t/63hLa4HVKLaH6Ou2/lZye4aZ30rG5vdzctm2bbDZb7XFeXp5uvPFG3XXXXS3OKygoaNCHvU+fPk7LEQAAAHCHI/tydfSTRRpSvEYpNfsy2Q2L8gJSVD3yPiVNnKFU/25Nzk1Oz5Bt0r3atTVL58oKFRDSVwlj0llxugxu/+QuLnoWLlyoAQMGaMKECS3OCw8PV69evRz6GhUVFaqoqKg9tlqtLYwGAAAA3Kfi/FnlfvKOAr5ersTKrxVTE/9eIdrX9zbF3jBHV8YnOPRa3j4+SrxmqvOS7WLcXjzVV1lZqRUrVuipp56SxdJyY/rk5GSdP39ew4YN07/9279p4sSJzY5dsGCBMjMz2ztdAAAAoN0cKtip4k8Xacix/9NonZJkrjLldr9K9uQMDZ84Xam+fm7OsmuzGIZhuDuJC95//33NnDlThw8fVnR0dJNjCgoK9NlnnyklJUUVFRVavny5Fi1apE2bNmn8+PFNzmlq5SkmJkbl5eUNLv0DAAAAXOn8uTPKW79c3fNWaFhlbm28VL31Xcydir/xIUX2H+TGDDs/q9Wq4OBgh2oDjyqe0tPT5efnp48+cmxjrwtuvfVWWSwWrVmzxqHxbfmAAAAAgPZ2cPdXKtn4uoaWrlWwzkiSbIZFuT3GSimzlDT+DvmwyuQSbakNPOayvUOHDmnDhg364IMP2jx37NixWrFihROyAgAAABxjq67WnouaM3jXa85w7swp5a1fpp67VmhoVb7iauIl6qMDsXfqiht/ppH9BrgldzjGY4qnJUuWKDw8XFOntv2GtpycHEVFRTkhKwAAAKB1OVlLFZ2dqUSdqI0dWx+qotR56tUvQaUbX9fQ43/TVTorSao2vJTbM03eo/9FieNuowNeB+ERf0t2u11LlixRRkaGfC46cZ5//nkVFhZq2bJlkqRXXnlFcXFxSkxMrG0wsXr1aq1evdodqQMAAKCLy8laqhFbHjMP6vU8CzdOKHzLY7JYpPiaWJElQodjf6yBkx9ScnSsy3PF5fGI4mnDhg06fPiwfvrTnzZ6rri4WIcPH649rqys1DPPPKPCwkIFBAQoMTFRa9eu1c033+zKlAEAAADZqqsVnW12dfa6qFn0hebRhiHt6DFOfmMfUOI10xTt7e3iLNFePKphhKvQMAIAAADtYdcXa5W4fmbr4258l/2WPFRbagMvF+UEAAAAdCp7d/5D9s0vOTT2XFmhk7OBK3jEZXsAAABAR3Cq/AflZy1WaMF7GmT7zuF5ASF9nZgVXIXiCQAAAGiBYbfr2x2bVP75m0oq+0RjLBWSpArDV7mB43XF6W3qZVgb3fMkSXZDKrWEKmFMuouzhjNQPAEAAABNKC87rj1Zb6rPtys1xH7QDFqkQ14xKh54txImz9bosEjlZC1Vry2PyW40bBphr+ksUJw6j1bknQR/iwAAAEANw25XwbYNOrVlsZJOfqoxlkpJ0nnDV7m9rldg2oMactUNivWqax2QnJ6hHEnR2ZmKqLfPU6klVMWp85ScnuHqtwEnodse3fYAAAC6vPITx7Q7601F7lupOPuR2vgBr1gdGzRDQ9NnK7h3nxZfw1ZdrT1bs3SurFABIX2VMCZd3qw4eby21Ab8bQIAAKBLMux27d6apbPZb2l4+WaNtVRJks4ZfsoNuUFB1z6oIaMmKt7LsQbV3j4+tCPv5CieAAAA0KWUfV+sgnVvKPq7VRpmr2khbpG+847X8SEzNXTyA7q6V6h7k4RHongCAABAp2fY7dqVvVYVX/6vhls/01hLtSTprOGvvN43qte4n2nQyHEa4OAqE7omiicAAAB0OI7eX3Ti2FHtzXpd/Q78SUlGsRm0SHu9B+qHhJlKTP+prg4KcXH26KgongAAANCh5GQtVXR2phLrdbY7tj5URTWd7ew2m3Z9sUZV/1yipFOfa6zFJkk6bQRoV1i6QsfP1qAR17orfXRgFE8AAADoMHKylmrElsfMg3p7KvUxTih8y2PatmuN+p7K1XDjWO2YAp8hKh86U4mTZ2lMYC+X54zOg+IJAAAAHYKtulrR2ZmSGm5Ge+HYMKSrrBskSaeMAOX3uUlhEx7SkOFjXZ0qOimKJwAAAHQIe7ZmmZfqWZp+3lITz466XyN/8luN6RHouuTQJdBOBAAAAB3C2RNHWh8kybfvcAVQOMEJWHkCAACARys5sk8H1i3SkCOrHBofENLXyRmhq6J4AgAAgMeprqpU7sb35ZWzTEln/6lIiyFJshkWecmovUSvPrshlVpClTAm3cXZoqugeAIAAIDHKDpYoEMbFmnA0b8oWT+YQYu0y2+Ezl35E0kWjdr2jAyjYdMIu1lbqTh1niKb2O8JaA+cWQAAAHCrqsoK5W1cKe+c5Uo695Wia1aZyhSkgshb1XfSQ0ocNKJ2fI6vn6KzMxVRb5+nUkuoimv2eQKcxWIYhuHuJFzNarUqODhY5eXlCgoKcnc6AAAAXVLh/t06vOE1DSr6q8J0sjae65+syhEZGj7pHvn5d2tyrq26Wnu2ZulcWaECQvoqYUy6vFlxwiVoS23AGQYAAACXqaw4r9xP3pPf10s1vCJHF1o7HFcv7Y2epv43zNHwKxJbfR1vHx8lXjPVuckCF6F4AgAAgNMd2Zerwg2vaXDJR0qRVZJkNyzKC0iRLfl+JU2coVQ/fzdnCbSM4gkAAABOUXH+rHI/eUcBXy9XYuXXiqmJf68Q7et3u2JvmKMr44a4NUegLSieAAAA0K4OFexU8aeLNOTY/2m0TkkyW4zndb9a9uT7NXzidKX6+rk5S6DtKJ4AAADQIkeaM5w/e1p5G5are947GlaZq9ia+DGFan/MHYq/8SGN6D/I9ckD7cjtxdP8+fOVmZnZIBYREaGSkpJm52zevFlPPfWUdu3apejoaD377LOaM2eOs1MFAADocnKylio6O1OJ9dqCH1sfqqKatuAHd3+lko2va2jpWo3WGUnmKtM3PVLlNTpDSeN/rAi64KGT8IgzOTExURs2bKg99vb2bnbsgQMHdPPNN2v27NlasWKFvvjiCz388MPq06eP7rzzTlekCwAA0CXkZC3ViC2PmQf1NqTtY5xQ+JbHdPjLFxVnFCquJl6iPjoQe6cGTJ6j5L7xrk4XcDqPKJ58fHwUGRnp0NhFixapf//+euWVVyRJQ4cO1VdffaWXX36Z4gkAAKCd2KqrFZ1tXh3kZWn43IXj/kahqg2LcnteI+/R/6LEcbcpklUmdGJe7k5Akvbu3avo6GjFx8drxowZ2r9/f7Njs7OzNXny5Aax9PR0ffXVV6qqqmpyTkVFhaxWa4MHAAAAmrdna5YidKJR4XSxb8a+ouT/t1ZXTvwxm9Si03N78TRmzBgtW7ZMWVlZevPNN1VSUqK0tDSdOHGiyfElJSWKiIhoEIuIiFB1dbWOHz/e5JwFCxYoODi49hETE9PkOAAAAJise79waJzdVunkTADP4fZfD9x00021/z18+HClpqZqwIABWrp0qZ566qkm51gsDX8FYhhGk/ELnn/++QavZbVaKaAAAAAuctpapl1Z/6veBe8ptXqvQ3MCQvo6OSvAc7i9eLpYjx49NHz4cO3d2/Q/2MjIyEad+EpLS+Xj46PQ0NAm5/j7+8vfnx2rAQAALmbY7dq78x86+fmbSjqxTmMsFZKkSsNbNnnLX5VNXrpnN6RSS6gSxqS7OGPAfTyueKqoqNDu3bs1bty4Jp9PTU3VRx991CC2bt06jR49Wr6+vq5IEQAAoMOznjyh3esWK6zgPQ221dxvbpEOe/VV0YC7NWTyz3RwxzqN2PKY7EbDphF286IfFafOo0EEuhS3n+3PPPOMbr31VvXv31+lpaX6zW9+I6vVqoyMDEnmJXeFhYVatmyZJGnOnDn6n//5Hz311FOaPXu2srOztXjxYr333nvufBsAAAAez7DbVbBjo6yfv6Wksk9qV5kqDF/lBl+n7qkPaOiYdPX3Mm+LD0nPUI6k6OxMRdTb56nUEqrimn2egK7E7cXT0aNHdc899+j48ePq06ePxo4dqy+//FKxsea+1MXFxTp8+HDt+Pj4eH388cd68skn9cc//lHR0dH67//+b9qUAwAANKO87Lj2ZL2p8G9XKsF+0AxapINeMSoZOEND02drdGhEk3OT0zNkm3Svdm3N0rmyQgWE9FXCmHRWnNAlWYwL3Ra6EKvVquDgYJWXlysoKMjd6QAAALQ7w25XwbYNOrXlLQ0/+am6WcwtXc4bvsrtNUmB1zyoIaMnyeLl9ubLgFu1pTbgVwYAAACdyMnjJdqz7k1F7VulBPsRM2iRDnjFqXTwDCVMflBX9e7j3iSBDoriCQAAoIMz7Hblf/l3nftysYaXb9bYmlWms4a/8kImKfja2Ro86jrFs8oEXBaKJwAAgA6q7PtiFWS9ruj97yvRXmgGLdJ33lfo+JB7NHTyA7q6V9NbuQBoO4onAAAAD2Krrtaei5ozeNdrzmC32ZSfvVYVW/9Xw63/0FhLtaSaVabQyQoZ9zMNHHGtBrDKBLQ7iicAAAAPkZO1VNHZmUqs1xb82PpQFaXOU8yI67V33RuKOfAnJRnF5pMWaa/PIJUlzNSwyf+iq4NC3JQ50DVQPAEAAHiAnKylGrHlMfOg3oa0fYwTCt/ymGxbvJRqsUuSThsB2hWWrtDxszVoxLVuyBbomiieAAAA3MxWXa3o7ExJkpel4XMXjn1kV4H3YFkTf6JhN2ZoTGAv1yYJgOIJAADA3fZszTIv1bO0PK76+vm66pqpLskJQGPcSQgAAOBGpYUHdH7L6w6NPVdW6ORsALSElScAAAAXq66qVN5nH0jb39bwM18q3GI4NC8gpK+TMwPQEoonAAAAFyk5vFcH1i/SFUc+1MgLHfUsUr5PoqKrDynION3onidJshtSqSVUCWPSXZswgAYongAAAJyouqpSuRvfl1fOUg0/u02RNatMZQpUQcQtirp+joYNGVnbbc9uNGwaYa9ZlCpOnadIH350A9yJf4EAAABOUHSwQIfWv6aBhX9RssrMoEXa5TdC50bcp+GT7tXYbt1rxyenZyhHUnR2piLq7fNUaglVceo8JadnuPgdALiYxTAMxy6y7USsVquCg4NVXl6uoKAgd6cDAAA6iarKCuV++p58di5X0rnt8qpZZfpBQfo28lb1nfSQYgaNaPE1bNXV2rM1S+fKChUQ0lcJY9LlzYoT4DRtqQ34lwgAAHCZCvfv0uENizSoaI1G6aQZtEi5/smqHJGh4ZPu0Vj/bg69lrePjxJpRw54JIonAACAS1BZcV65n7wjv6+Xa3hFji70wTuuXtob/SP1v+EhDb8i0a05AmhfFE8AAABtcGTv1yr85HUNLvlIKbJKkuyGRXkBo2VLvk9JE2co1c/fzVkCcAaKJwAAgFacP3dGeZ+8o4Bvliux8hvF1MS/V4j29btdsTfM0ZVxQ9yaIwDno3gCAABdVmvNGQ7t2aHija9ryLG1Gq1T5hzDorzuV8sYNUtJ1/1Yqb5+7kofgItRPAEAgC4pJ2uporMzlVivLfix9aE6MvoXkt2m7rkrNKwqT7EXnlOo9sfcofgbH9KI/oPckzQAt6J4AgAAXc6FDWklSfU2pA03Tih82/+TpSZmMyz6pkeqvEZnKGn8jxVBy3CgS+M7AAAA6FJs1dWKzs6UJHlZGj53oWiqNrz0z/4PauCUh5XcN97FGQLwVF7uTgAAAMCV9mzNUoRONCqc6vOx2BWccJ3CKZwA1MPKEwAA6BLOnDqpXeveVnTeaw6NP1dW6OSMAHQ0FE8AAKBT2/f15zrx2ZtKPJ6lqy3nHJ4XENK39UEAuhSKJwAA0OmctpZpV9b/qveedzXItk8DJckiHbVE6XDcnRp04B2FGmVNXrpnN6RSS6gSxqS7Om0AHs7t9zwtWLBAV111lQIDAxUeHq7bbrtNBQUFLc7ZtGmTLBZLo8eePXtclDUAAPA0ht2ub3ds1j//6155/ecQjdn1aw2y7VOl4aPtgdcr78YViv63XUrLeEFHU82GEXaj4WtcOC5OnddgvycAkDxg5Wnz5s165JFHdNVVV6m6ulq//OUvNXnyZOXn56tHjx4tzi0oKFBQUFDtcZ8+fZydLgAA8DDWkye0O+sthX27UoNt+82gRTrs1VdFA+7WkMk/U0qfqAZzktMzlCMpOjtTEfX2eSq1hKo4dZ6S0zNc+A4AdBQWwzCM1oe5zvfff6/w8HBt3rxZ48ePb3LMpk2bNHHiRJWVlalXr15t/hpWq1XBwcEqLy9vUHwBAICOwbDbVbBjo6yfv6Wksk/U3VIhSaowfJUbfJ26pz6goWPSZfFq+SIbW3W19mzN0rmyQgWE9FXCmHRWnIAupi21gcd9dygvL5ck9e7du9WxycnJOn/+vIYNG6Z/+7d/08SJE5scV1FRoYqKitpjq9XaPskCAACXKv/he+3OelMRe1cqwX7IDFqkg14xKhk4Q0PTZ2t0aITDr+ft46PEa6Y6KVsAnY1HFU+GYeipp57Stddeq6SkpGbHRUVF6Y033lBKSooqKiq0fPlyTZo0SZs2bWpytWrBggXKzMx0ZuoAAMBJDLtdBds26NSWtzT85Kcaa6mSJJ03fJXba5ICr3lQQ0ZPUlwrq0wAcLk86rK9Rx55RGvXrtXnn3+ufv36tWnurbfeKovFojVr1jR6rqmVp5iYGC7bAwDAg508XqI9695U1L5VirUfqY0f8IpT6eAZSpj8oIJ7c78zgMvTIS/be/TRR7VmzRp99tlnbS6cJGns2LFasWJFk8/5+/vL39//clMEAACXwZH7iwy7Xflf/l3nvlys4eWba1eZzhr+yguZpOBrZ2vwqOsUzyoTADdwe/FkGIYeffRRffjhh9q0aZPi4+Mv6XVycnIUFRXV+kAAAOByOVlLFZ2dqcR6ne2OrQ9VUU1nu7Lvi1WQ9bqi97+vRHuhOcAifed9hY4Pmalh6Q/o6uDW74cGAGdye/H0yCOP6N1339Vf//pXBQYGqqSkRJIUHBysgIAASdLzzz+vwsJCLVu2TJL0yiuvKC4uTomJiaqsrNSKFSu0evVqrV692m3vAwAANC0na6lGbHnMPKi3KW0f44TCtzym3dsWaUDVtxprqZYknTG6KS90snqPm62BI67VAFaZAHgItxdPr732miTpuuuuaxBfsmSJZs2aJUkqLi7W4cOHa5+rrKzUM888o8LCQgUEBCgxMVFr167VzTff7Kq0AQCAA2zV1YrONps2eVkaPnfheGh1vmSR9voMUlnCTA2b/C8aExTi4kwBoHUe1TDCVdjnCQAA19j1xVolrp/Z6ritif+uMXc97YKMAKChttQGrIMDAACnOX00z6Fx3gGBTs4EAC6f2y/bAwAAnYutulp5//hQ9q/e1ujTXzS4z6k5ASF9nZ8YAFwmiicAANAuSgsP6Lt1ixR/aLVG6HszaJEqDR/5qlqWJooouyGVWkKVMCbdtckCwCWgeAIAAJesuqpSeZ99IG1/W8PPfKlwi3krdbl6aHf4VEVOfEhlh3dpxJbHZBgNm0bYa+66Lk6dp0gffiQB4Pn4TgUAANqs5PBeHVi/SFcc+VAjL+zdZJHy/YbrbNJPlHTjfRob0EOSFDd0tHIkRWdnKqLePk+lllAV1+zzBAAdAd326LYHAIBDqqsqlbvxfXnlLNXws9vkVbPKVKZAFUTcoqjr5yh2yMhm59uqq7Vna5bOlRUqIKSvEsaky5sVJwBu1pbagO9YAACgRUUHC3Ro/WsaWPgXJavMDFqkPP+RqrjyPiVNmqmx3bq3+jrePj5KvGaqk7MFAOeheAIAAI1UVVYo99P35LNzuZLObVd0zSrTDwpSQdQ09Zs0R0kDh7s5SwBwLYonAABQq3D/Lh3esEiDitZolE6aQYuU6z9KVSPvV9L19yjVv5tbcwQAd6F4AgCgi6usOK/cT96R/9fLlFSxUxd2XDquXtob/SP1v+HnGn7FULfmCACegOIJAIBOxtHGDEf2fq3CT17X4JKPlCKrJMluWJQXMFrVyRkaPnG6Uv38XZ0+AHgsiicAADqRnKylis7OVGK9luDH1oeqqKYl+PlzZ5T3yTsK+Ga5Eiu/UUzNmFL11v5+t6v/DQ/pyrgh7kkeADwcxRMAAJ1ETtZSjdjymHlQbzPaPsYJhW95TDt3vqvYc7s0WqckSTbDorzuV8sYNUtJ1/1Y4b5+bsgaADoOiicAADoBW3W1orMzJUlelobPXTgeee5LSdIxhWp//zsVf+NDGhEz0JVpAkCHRvEEAEAnsGdrlnmpnqXlcdlxj+jqn/xaEWxOCwBt5uXuBAAAwOU7W7rfoXG+YXFNNo8AALSO754AAHRg3+V+qeObX1fS92tbXXWSpICQvq0PAgA0ieIJAIAO5sypk9q17m312v2OBld/qwGSZJGqDS95yy5LE0WU3ZBKLaFKGJPu6nQBoNOgeAIAoIPY9/XnOvHZG0o8vk5XW85JkqoMb+UGXivfqx9Q5ekyJW99XIbRsGmE3TD/LE6dp0gu2QOAS8Z3UAAAPNip8h+Uv26Jeu95V4Ns+zRQkizSUUuUjsbfpYGTf6ZRkTG143O8LYrOzlREvX2eSi2hKq7Z5wkAcOkshmEY7k7C1axWq4KDg1VeXq6goCB3pwMAQAOG3a69O/+hk/94Q0k/rFd3S4UkqdLwUW7QePmP/amGjb1ZXt7eTc63VVdrz9YsnSsrVEBIXyWMSadJBAA0oy21Ad9JAQDwENaTJ7Q76y2FfbtSg2013fMs0mGvvioacLeGTP6ZUvpEtfo63j4+SrxmqpOzBYCuh+IJAAA3Mux2FWz/VNYvFmt42QaNsVRKkioMX+UGX6fuqQ9o6Jh09fdidxEAcDeKJwAA3KD8h++1O+tNRexdqQT7ITNokQ56xahk4AwNTZ+t0aER7k0SANAAxRMAAC5i2O3as229Tm95S8NPbtRYS5Uk6bzhq9xekxR4zYMaMnqS4lhlAgCPRPHkZja7oX8e+EGlp84rPLCbro7vLW8vB3Y5JCdyIidyIifPyMmB5gwnj5dod9abiv5ulYbaj5hBi3TAK06lg2coYfKDuqp3HzdkDwBoC48onl599VX9x3/8h4qLi5WYmKhXXnlF48aNa3b85s2b9dRTT2nXrl2Kjo7Ws88+qzlz5rgw4/bx97xiZX6Ur+Ly87WxqOBumnfrME1Jav2GYHIiJ3IiJ3Jyb045WUsVnZ2pxHptwY+tD1VR6jyNvPE+5X/5d537crGGl29Was0q01nDX3khkxR87WwNHnWd4lllAoAOw+2tyletWqX77rtPr776qq655hq9/vrreuutt5Sfn6/+/fs3Gn/gwAElJSVp9uzZeuihh/TFF1/o4Ycf1nvvvac777zToa/pCa3K/55XrJ+v2KGLP/wLvz997SejXP7DADmREzmREzk5LidrqUZseUxS4w1pLZK+t4QoXGW18e+8r9DxIfdo6OQHFNQr1KW5AgCa15bawO3F05gxYzRq1Ci99tprtbGhQ4fqtttu04IFCxqNf+6557RmzRrt3r27NjZnzhx9/fXXys7Oduhrurt4stkNXfvipw1+e3qx8EB/rfpZqssuR7HZDU1/PVvfn64gJ3IiJ3Iip9Zyqq5W99dGqo9xQi19yTOGn3aFpitk3M80cMS1srDKBAAep8MUT5WVlerevbv+9Kc/6fbbb6+NP/7449q5c6c2b97caM748eOVnJys//qv/6qNffjhh5o+fbrOnj0rX1/fRnMqKipUUVH3P12r1aqYmBi3FU/Z353QPW9+6fKvCwBoH2O98rXS7zetjts5/k2NvH66CzICAFyqDrNJ7vHjx2Wz2RQR0bAVa0REhEpKSpqcU1JS0uT46upqHT9+XFFRjS/bWLBggTIzM9sv8ctUeqr5Faf6/L0t8vF2zW8pq212Vdhar6PJiZwcQU6OISfHeFpOFtl1nfGNQ2Orz550bjIAAJfyiIYRFkvDax4Mw2gUa218U/ELnn/+eT311FO1xxdWntwlPLCbQ+Pe/ukYpQ5wzXXxjq6GkRM5OYKcHENOjvGYnE6VSDkrpB3LpJOHHJoSENLXefkAAFzOrRdfh4WFydvbu9EqU2lpaaPVpQsiIyObHO/j46PQ0Kb/p+nv76+goKAGD3e6Or63ooK7qbny0CKzg9TV8b3JiZzIiZzIyZ052W3S3vXSynul3w2TPv3/pJOHZPgF6qy6yd7MgpjdkEoUqoQx6e2fEwDAbdxaPPn5+SklJUXr169vEF+/fr3S0tKanJOamtpo/Lp16zR69Ogm73fyRN5eFs27dZgkNfph4MLxvFuHuXTvEnIiJ3IiJ3Kqx1okbX5J+q8R0js/lvb8n2TYpJix0m2vyfLMtypIfUmSGhVQF46LU+c12u8JANCxub3b3oVW5YsWLVJqaqreeOMNvfnmm9q1a5diY2P1/PPPq7CwUMuWLZNU16r8oYce0uzZs5Wdna05c+Z0uFblkmfuWUJO5ERO5NRlc7JVS/s2SNvflvZmSYbdjHfrJY24R0rJkMKHNphyYZ+niHr7PJUoVMWp85ScnnH5OQEAnK7DdNu74NVXX9VLL72k4uJiJSUl6fe//73Gjx8vSZo1a5YOHjyoTZs21Y7fvHmznnzyydpNcp977rk2bZLrKcWTZLbg/eeBH1R66rzCA83LTlz5G11yIidyIqcun9PJI1LOcvN+JmthXTz2GillljR0muTb/L2qtupq7dmapXNlhQoI6auEMemsOAFAB9LhiidX86TiCQDgBrZqc3Vp+9vmPU0Xtt8N6C2NnCmNypD6DHZnhgAAF+kwrcoBAHCpskNmt7ycFdLpes2H4sbVrDLdKvn4uy09AIBno3gCAHRutiqp4GNp+1Lpu09Vu8rUPUxKvtdcZQod4NYUAQAdA8UTAKBz+mF/zSrTO9KZ0rr4FRPNVaYhN0s+fm5LDwDQ8VA8AQA6DrtNOrRFOn1M6hkhxaZJXt51z1dXmm3FdyyV9m+qi/eMkEbeK426X+od7/K0AQCdA8UTAKBjyF8j/f05cw+mC4KipSkvSuHDzIJp57vS2eM1T1qkgZPMVabBUyTvjrEXIADAc1E8AQA8X/4a6f37VXu/0gXWIun9+xrGAqOk5PukUfdJvfq7LEUAQOdH8QQA8Gx2m7nidHHhdLFB6eYq06DJkjf/ewMAtD/+7wIA8GyHtjS8VK85aY9K8eOcnw8AoMvycncCAAA061i+9PnvHRt7+phzcwEAdHmsPAEAPEvlWWnXh9L2t6Wj/3R8Xs8Ip6UEAIBE8QQA8BQlueZGtt+8L1WUmzEvH7NT3uFs6ewPavq+J4vZdS82zZXZAgC6IIonAID7VJyWdn1grjIVbq+Lh8RJozLMvZkCI+p127OoYQFlMf+YsrDhfk8AADgBxRMAwPWKdpoFU+6fpcpTZszLVxp6i1k0xU+QvOrdljtsmjR9WTP7PC00nwcAwMkongAArnHeKuX92Syair+ui/e+wmwxPmKm1LNP8/OHTZMSpprd904fM+9xik1jxQkA4DIUTwAA5zEMqWhHzSrTaqnqjBn39pOGTjOLprhrJYvFsdfz8qYdOQDAbSieAADt73y52fhh+1LpWG5dPGyweVneiHukHqHuyw8AgEtA8QQAaB+GIR3dZq4y5X0gVZ8z497+UuJt5ipT/1THV5kAAPAwFE8AgKbZbY7dX3SuTPp6lbRjqVSaXxfvM9QsmK6cLnXv7bK0AQBwFoonAEBj+Wua6Wz3otm4wTCkw1+aq0z5f5Gqz5tjfAKkpDvMS/NirmaVCQDQqVA8AQAaqt1T6aINaa3FZnzkPdLR7dLxgrrnIpLMVabhd0kBvVyYLAAArkPxBACoY7eZK04XF05SXWznu+afvt2lpDullH+R+o5ilQkA0OlRPAEA6hza0vBSveaMfVi67nmpW5DzcwIAwEN4tT4EANBlnCp2bFzfFAonAECXw8oTAEA6XSrtfEf6cpFj43tGODcfAAA8EMUTAHRVdru0f6PZMa/gY8leXfOERU3f81TzXFC02bYcAIAuhuIJALoaa7G0c4W0Y5l08nBdvN9VZsc8b3/pg9k1wfpFVE1DiCkLm97vCQCATs5t9zwdPHhQDzzwgOLj4xUQEKABAwZo3rx5qqysbHHerFmzZLFYGjzGjh3roqwBoIOy26S966WV90q/T5Q+/Y1ZOPkHS1f/TJrzhfTgBin5J9KVd0nTl0lBUQ1fIyjajA+b5p73AACAm7lt5WnPnj2y2+16/fXXNXDgQOXl5Wn27Nk6c+aMXn755RbnTpkyRUuWLKk99vPzc3a6ANAxlRdKOSuknOVS+ZG6eMxYc5Vp2I8kv+6N5w2bJiVMNbvvnT5m3uMUm8aKEwCgS3Nb8TRlyhRNmTKl9viKK65QQUGBXnvttVaLJ39/f0VGRjo7RQDomGzV0r715r1Me9dJht2Md+sljZwpjcqQwhNafx0vbyl+nDMzBQCgQ/Goe57Ky8vVu3fvVsdt2rRJ4eHh6tWrlyZMmKAXXnhB4eHhzY6vqKhQRUVF7bHVam2XfAHAo5w8Yq4w7Vgunaq3V1PstVJKhjR0muTbzX35AQDQwXlM8fTdd9/pD3/4g/7zP/+zxXE33XST7rrrLsXGxurAgQP61a9+peuvv17bt2+Xv79/k3MWLFigzMxMZ6QNAO5lq5K+zTJXmfZtUG2Dh4DedatMfQa7M0MAADoNi2EYzfWjvSTz589vtVDZtm2bRo8eXXtcVFSkCRMmaMKECXrrrbfa9PWKi4sVGxurlStX6o477mhyTFMrTzExMSovL1dQEJs8AuiAyg6a3fJy3pFOl9TF48eb9zIl3CL5NP0LJQAAUMdqtSo4ONih2qDdV57mzp2rGTNmtDgmLi6u9r+Lioo0ceJEpaam6o033mjz14uKilJsbKz27t3b7Bh/f/9mV6UAwCPYba03Z7BVmfsxbX9b+m6jaleZuodJyfeaq0yhA1ydOQAAXUa7F09hYWEKCwtzaGxhYaEmTpyolJQULVmyRF5ebe+cfuLECR05ckRRUVGtDwYAT5S/Rvr7c5K13n1KQdHSlBfNrncnvjNXmXa+I535vm7MFRPNVaYhN0s+dB0FAMDZ2v2yPUdduFSvf//+WrZsmby9637DWr+TXkJCghYsWKDbb79dp0+f1vz583XnnXcqKipKBw8e1L/+67/q8OHD2r17twIDAx362m1ZmgMAp8pfI71/vxpuRiuZG9IaUvgwqTS/LtwzQhp5rzTqfql3vAsTBQCgc3LrZXuOWrdunfbt26d9+/apX79+DZ6rX88VFBSovLxckuTt7a3c3FwtW7ZMJ0+eVFRUlCZOnKhVq1Y5XDgBgMew28wVp0aFk+piFwqngTeYq0yDp0jevi5KEAAA1Oe2lSd3YuUJgEc48A9p6S2tj7tzsTT8x87PBwCALqgttUHbbzICALSPohx3ZwAAANrAY/Z5AoAuoeqclP9Xs2Pe4WzH5vSMcGpKAADAMRRPAOAKx/LNgumbldL58pqgl9klr/p8M5MsZte92DQXJQkAAFpC8QQAzlJ5Vtr1oVk0Hf1nXTy4v9ktL/le6ehXNd32pIaNIyzmH1MWNt7vCQAAuAXFEwC0t5LcmlWm96UKqxnz8pGG3GR2zLtiYl1BNGyaNH1ZM/s8LTSfBwAAHoHiCQDaQ8VpKW+1tGOpVLi9Lh4SJ43KMPdmCmzm3qVh06SEqdKhLdLpY+Y9TrFprDgBAOBhKJ4A4HIU7TRXmXL/JFWeNmNevtLQW8yiKX6C5OVAY1Mvbyl+nDMzBQAAl4niCQDa6rxVyvuzWTQVf10X7z1ASsmQRsyUevZxW3oAAMA5KJ4AwBGGIRXtqFllWi1VnTHj3n7S0GnmvUxx10oWizuzBAAATkTxBAAtOV9uNn7YvlQ6llsXDxtsFkxXzpB6hLotPQAA4DoUTwC6Hrut5eYMhiEd3WYWTHmrpepzZtzbX0q8zSya+qeyygQAQBdD8QSga8lf00xb8BfNhg3fvG9emleaX/d8n6E1q0zTpe69XZ0xAADwEBRPALqO/DU1G9IaDePWIun9+8wuefYqM+YTICXdYRZN/a5ilQkAAFA8Aegi7DZzxeniwqnBmCopPFEa/S/S8LukgF6uyg4AAHQAFE8AuoZDWxpeqtecmxZK8eOdnw8AAOhwHNi5EQA6uDPHzeYPjjhd6txcAABAh8XKE4DOyW6XDn5mFk27P6q7l6k1PSOcmxcAAOiwKJ4AdC6nS6Wd75hFU9mBunhUsnl8vlxN3/dkMbvuxaa5KlMAANDBUDwB6Pjsdmn/RrPFeMHHkr3ajPsHme3FR2VIUVfW67ZnUcMCqqaT3pSFDfd7AgAAqIfiCUDHZS2Wdq6QdiyXTh6qi/e7WkrJkBJvl/x61MWHTZOmL2tmn6eF5vMAAADNoHgC0LHYbdJ3n9asMv1NMmxmvFuwdOUMs2iKSGx+/rBpUsJUs/ve6WPmPU6xaaw4AQCAVlE8AegYygulnBVSznKp/EhdvH+quZHtsB9JvgGOvZaXtxQ/zilpAgCAzoviCYDnslVL+9abzR/2ZkmG3YwHhEgj7jHvZQpPcG+OAACgy6B4AuB5Th4xV5h2LJdO1bs3KfZac5Vp6K2Sbze3pQcAALomiicAnsFWJX2bZd7LtG+DarvhdQ81V5lSZklhg9yYIAAA6OoongA4l93WcnOGsoPSjmVSzjvS6ZK6ePx4s2BKuEXy8Xd11gAAAI24tXiKi4vToUOHGsSee+45LVy4sNk5hmEoMzNTb7zxhsrKyjRmzBj98Y9/VGJiC921ALhH/pqm24JPfkGyeEk7lkrfbVTtKlOPPtLIe6VR90uhA9ySMgAAQHPcvvL061//WrNnz6497tmzZ4vjX3rpJf3ud7/T22+/rcGDB+s3v/mNbrzxRhUUFCgwMNDZ6QJwVO2GtEbDuLVI+vO/NIxdMdFcZRpys+Tj56oMAQAA2sTtxVNgYKAiIyMdGmsYhl555RX98pe/1B133CFJWrp0qSIiIvTuu+/qoYcecmaqABxlt5krThcXTvVZvKRrHjc75vWOd1lqAAAAl8rL3Qm8+OKLCg0N1ciRI/XCCy+osrKy2bEHDhxQSUmJJk+eXBvz9/fXhAkTtGXLlmbnVVRUyGq1NngAcKJDWxpeqtcUwy4NmEThBAAAOgy3rjw9/vjjGjVqlEJCQvTPf/5Tzz//vA4cOKC33nqryfElJebN5BEREQ3iERERje6dqm/BggXKzMxsv8QBNK3qvLT7I+mz/3Bs/Oljzs0HAACgHbX7ytP8+fNlsVhafHz11VeSpCeffFITJkzQlVdeqQcffFCLFi3S4sWLdeLEiRa/hsViaXBsGEajWH3PP/+8ysvLax9Hjhy5/DcKoE7pHunvz0u/S5A+eFA6XuDYvJ4RrY8BAADwEO2+8jR37lzNmDGjxTFxcXFNxseOHStJ2rdvn0JDQxs9f+HeqJKSEkVFRdXGS0tLG61G1efv7y9/f1odA+2q6pyU/1dzX6bD2XXxoH5S8k+k7Uuk06Vq+r4ni9l1LzbNRckCAABcvnYvnsLCwhQWFnZJc3NyciSpQWFUX3x8vCIjI7V+/XolJydLkiorK7V582a9+OKLl5YwgLY5tkvavlT6ZqV0vtyMWbylwVPMjnkDJ5n7OEUk1nTbs6hhAVWzSjxlYcP9ngAAADyc2+55ys7O1pdffqmJEycqODhY27Zt05NPPqlp06apf//+teMSEhK0YMEC3X777bJYLHriiSf029/+VoMGDdKgQYP029/+Vt27d9fMmTPd9VaAzq/yjLTrQ3OV6ei2unhwfynlfmnkT6Sgi37pMWyaNH1Z0/s8TVloPg8AANCBuK148vf316pVq5SZmamKigrFxsZq9uzZevbZZxuMKygoUHl5ee3xs88+q3Pnzunhhx+u3SR33bp17PEEOEPxN+ZGtt+8L1XUdKn08jH3Y0rJkK64XvJq4dbJYdOkhKlm973Tx8x7nGLTWHECAAAdksUwjBY2YumcrFargoODVV5erqCgIHenA3iWitNS3mpzlaloR108JM7ck2nkvVIgjR4AAEDn0JbawO2b5ALwEEU55r1MuX+SKk+bMS9faegt5r1MceNbXmUCAADo5CiegK7svFXK+7O5ylT8dV289wCzYBpxj9Szj7uyAwAA8CgUT0BnYre1fn+RYUiFO8xW4nkfSFVnzLi3nzTsR+aleXHXSi3snQYAANAVUTwBnUX+mmY6271oNm44X242fti+VDqWWzcmbLC5ynTlDKlH4/3VAAAAYKJ4AjqD/DU1eypd1P/FWiy9f58UN046+pVUfc6Me/tLibebRVP/sawyAQAAOIDiCejo7DZzxeniwkmqix38h/ln+DDzsrwrp0vde7sqQwAAgE6B4gno6A5taXipXnNuflm66kFWmQAAAC4RfYeBju7Ed46NCwihcAIAALgMrDwBHZFhSAc/N1uM5//FsTk92dgWAADgclA8AR3J6e+lr9+VdiyTTuyri3v5SvaqZiZZzK57sWkuSREAAKCzongCPJ3dLh38zFxl2v1/dUWSX09p+F1SSoZ08khNtz2pYeOImsv0pixsvN8TAAAA2oTiCfBUp45JO9+RdiyVyg7WxfummC3GE++Q/HuasehkafqyZvZ5Wmju8wQAAIDLQvEEeBK7Xdr/qbmRbcHHkr3ajPsHme3FR2VIUVc2PXfYNClhqtl97/Qx8x6n2DRWnAAAANoJxRPgCazF0s4V5r1MJw/XxftdXbPKdJvk16P11/HyluLHOStLAACALo3iCXAXu03a94l5L9O3f5cMmxnvFixdOcO8lyki0a0pAgAAoA7FE+Bq5UelnBXSjuWS9WhdvH+quco07EeSb4Db0gMAAEDTKJ4AV7BVS/vWm6tMe9dJht2MB4RII2ZKo+6XwhPcmiIAAABaRvEEXCq7rfXmDCcPmytMOculU8V18dhrzVWmobdKvt1cmjYAAAAuDcUTcCny1zTTFvxFachN5j1M298272m6sO9S91Bp5EyzY17YIHdkDQAAgMtA8QS0Vf6amg1pjYZxa7H0/n1mw4fz5XXx+Alm84eEWyQff5emCgAAgPZD8QS0hd1mrjhdXDhJdbHz5VL3MCn5J+a9TKEDXJkhAAAAnITiCWiLQ1saXqrXnDvekgZOdH4+AAAAcBkvdycAdBjVFdLuNY6NPXfCubkAAADA5Vh5AlpzfK/Z/GHnu9K5Hxyb0zPCqSkBAADA9SiegKZUnTdXmba/LR36oi7eM0qqPG0+mrzvyWJ23YtNc1GiAAAAcBWKJ6C+0j3SjqXS1+9J58rMmMVLGjTZ3Jdp4I1Swcc13fYsalhAWcw/pixsvN8TAAAAOjy33fO0adMmWSyWJh/btm1rdt6sWbMajR87dqwLM0enU3VO2vmetDhdenWM9OWrZuEU1E+67l+lJ/KkmavM/Zu8faRh06Tpy6SgqIavExRtxodNc8/7AAAAgFO5beUpLS1NxcXFDWK/+tWvtGHDBo0ePbrFuVOmTNGSJUtqj/38/JySIzq5Y7vMy/K+WVW3L5PF2yySRmVIAyc1v4I0bJqUMNXsvnf6mHmPU2waK04AAACdmNuKJz8/P0VGRtYeV1VVac2aNZo7d64sFkuLc/39/RvMbU1FRYUqKipqj61Wa9sTRudQeUba9aFZNB2tt8IZ3F9KuV8a+ZPGK0rN8fKW4sc5JU0AAAB4Ho+552nNmjU6fvy4Zs2a1erYTZs2KTw8XL169dKECRP0wgsvKDw8vNnxCxYsUGZmZjtmiw6n+BuzYMr9k1RRUzx7+UhDbjbvZbpiouRF534AAAA0z2IYRlMtw1zu5ptvliR9/PHHLY5btWqVevbsqdjYWB04cEC/+tWvVF1dre3bt8vf37/JOU2tPMXExKi8vFxBQUHt9ybgWSpOSXmrzaKpKKcuHhIvpWRII++VejZfdAMAAKDzs1qtCg4Odqg2aPeVp/nz57e6yrNt27YG9zUdPXpUWVlZev/991t9/bvvvrv2v5OSkjR69GjFxsZq7dq1uuOOO5qc4+/v32xhhU6oKKdmlenPNS3FJXn5SkNvMVeZ4sazygQAAIA2a/fiae7cuZoxY0aLY+Li4hocL1myRKGhoZo2re1dyqKiohQbG6u9e/e2eS46ELut5eYM563mJXnb35ZKvqmL9x5gFkwjZ0o9wlydNQAAADqRdi+ewsLCFBbm+A+phmFoyZIluv/+++Xr69vmr3fixAkdOXJEUVEO3uSPjid/jfT35yRrUV0sKNrcTymor1kw5a2Wqs6az3n7ScN+ZHbMi7tWaqUBCQAAAOAItzeM+PTTT3XgwAE98MADTT6fkJCgBQsW6Pbbb9fp06c1f/583XnnnYqKitLBgwf1r//6rwoLC9Ptt9/u4szhEvlrajakvejWPGtRTbyesMHmKtOVM6Qeoa7KEAAAAF2E24unxYsXKy0tTUOHDm3y+YKCApWXm3vweHt7Kzc3V8uWLdPJkycVFRWliRMnatWqVQoMDHRl2nAFu81ccbq4cLrY8OnS6J9K/ceyygQAAACncXvx9O6777b4fP1mgAEBAcrKynJ2SvAUh7Y0vFSvOaPul2JTnZ8PAAAAujS3F09AI4YhHc6WPv2NY+NPH3NuPgAAAIAonuBJzpyQvn5P2rFUOv6t4/N6RjgvJwAAAKAGxRPcyzCkg/+Qti+Vdq+RbJVm3LeHlHS79G2WdOa4mr7vyWJ23YtNc2XGAAAA6KIonuAep7+Xvn7XLJp++K4uHjXC7JiX9GOpW1C9bnsWNSygahpDTFnYcL8nAAAAwEkonuA6drt0YLO5L9OetZK9yoz79ZSG3yWlZEjRyQ3nDJsmTV/W/D5Pw9q+sTIAAABwKSie4Hynjkk73zHvZSo7WBfvm2KuMiXeIfn3bH7+sGlSwlSz+97pY+Y9TrFprDgBAADApSie4Bx2u7T/U3OVqeBvkr3ajPsHSVdOl0ZlSFFXOv56Xt5S/DinpAoAAAA4guIJ7ctaLO1cIe1YJp08XBfvd3XNKtNtkl8Pd2UHAAAAXDKKJ1w+u03a94m5yvTt3yXDZsa7BUsj7jFXmSKGuTVFAAAA4HJRPOHSlR+VclZIO5ZL1qN18f5pZvOHYT+SfAPclx8AAADQjiie0Jjd1nxzBlu1tG+9ucq0d51k2M14QIg0YqZZNPUZ4rbUAQAAAGeheEJD+Wuabgs+/v+Z9zPlLJdOFdc9FzfOvJcp4RbJt5vL0wUAAABcheIJdWo3pDUaxq1F0v89WXfcPVQaea95L1PYQJemCAAAALgLxRNMdpu54nRx4VSft79026vS0FslH3+XpQYAAAB4Ai93JwAPsf+zhpfqNcVWYd4DReEEAACALoiVp67uxHfSjqXSV//r2PjTx5ybDwAAAOChKJ66ouoKac//mR3zDnzWtrk9I5ySEgAAAODpKJ66kuN7zYLp6/eksydqghZp0I1S8v3S356t6aTX1H1PFrPrXmya6/IFAAAAPAjFU2dXdV7a/ZFZNB36vC4eGC2Nul9K/onUK6YmaNR027OoYQFlMf+YsrBuvycAAACgi6F46qxK95j3Mn39nnSuzIxZvKRB6ea+TANvkLwv+usfNk2avqzpfZ6mLDSfBwAAALooiqfOpOqctOsv5irTkS/r4sEx5irTyHul4L4tv8awaVLCVOnQFrM5RM8I81I9VpwAAADQxVE8dQbHdpkF0zerpPPlZsziLQ25yVxlGnB924ofL28pfpwzMgUAAAA6LIqnjqryjLTrQ7NoOrqtLt6rvzQqw7yXKTDSbekBAAAAnQ3FU0dT/I1ZMOX+SaqwmjEvH/NSu1EZ0hUTJS/2PgYAAADaG8WTu9ltrd9fVHFaylttFk1FO+riIfHmZXkjZ0o9w12ZNQAAANDlUDy5U/6aZjrbvWg2bijKqVll+rNUedp83stXGnqrWTTFjWOVCQAAAHARp/7k/cILLygtLU3du3dXr169mhxz+PBh3XrrrerRo4fCwsL02GOPqbKyssXXraio0KOPPqqwsDD16NFD06ZN09GjR53wDpwof425p1L9wkmSrMXS+/dJr1wpvXGdWTxVnpZCB0qTfyM9vUe6a4l0xQQKJwAAAMCFnPrTd2Vlpe666y79/Oc/b/J5m82mqVOn6syZM/r888+1cuVKrV69Wk8//XSLr/vEE0/oww8/1MqVK/X555/r9OnTuuWWW2Sz2ZzxNtqf3WauODXYiPaCmtjJQ5KXnzT8LmnWWmnuV1Lao1KPMFdmCgAAAKCGxTCMpn6Cb1dvv/22nnjiCZ08ebJB/G9/+5tuueUWHTlyRNHR0ZKklStXatasWSotLVVQUFCj1yovL1efPn20fPly3X333ZKkoqIixcTE6OOPP1Z6enqjORUVFaqoqKg9tlqtiomJUXl5eZNfw+kO/ENaekvr42a8JyXc7Px8AAAAgC7KarUqODjYodrArdd9ZWdnKykpqbZwkqT09HRVVFRo+/btTc7Zvn27qqqqNHny5NpYdHS0kpKStGXLlibnLFiwQMHBwbWPmJiY9n0jbXX6mGPjqs46Nw8AAAAADnNr8VRSUqKIiIgGsZCQEPn5+amkpKTZOX5+fgoJCWkQj4iIaHbO888/r/Ly8trHkSNH2ucNXKqeEa2Pacs4AAAAAE7X5uJp/vz5slgsLT6++uorh1/PYrE0ihmG0WS8JS3N8ff3V1BQUIOHW8WmmV311Nx7tEhBfc1xAAAAADxCm1uVz507VzNmzGhxTFxcnEOvFRkZqa1btzaIlZWVqaqqqtGKVP05lZWVKisra7D6VFpaqrS0DlJseHmb7cjfv19mAVX/trOagmrKwsb7PQEAAABwmzavPIWFhSkhIaHFR7du3Rx6rdTUVOXl5am4uLg2tm7dOvn7+yslJaXJOSkpKfL19dX69etrY8XFxcrLy+s4xZNk7uM0fZkUFNUwHhRtxodNc09eAAAAAJrk1E1yDx8+rB9++EGHDx+WzWbTzp07JUkDBw5Uz549NXnyZA0bNkz33Xef/uM//kM//PCDnnnmGc2ePbv20rrCwkJNmjRJy5Yt09VXX63g4GA98MADevrppxUaGqrevXvrmWee0fDhw3XDDTc48+20v2HTpISp0qEtZhOJnhHmpXqsOAEAAAAex6nF07//+79r6dKltcfJycmSpI0bN+q6666Tt7e31q5dq4cffljXXHONAgICNHPmTL388su1c6qqqlRQUKCzZ+s6z/3+97+Xj4+Ppk+frnPnzmnSpEl6++235e3dAYsOL28pfpy7swAAAADQCpfs8+Rp2tLLHQAAAEDn1WH2eQIAAACAjoLiCQAAAAAcQPEEAAAAAA6geAIAAAAAB1A8AQAAAIADnNqq3FNdaDBotVrdnAkAAAAAd7pQEzjShLxLFk+nTp2SJMXExLg5EwAAAACe4NSpUwoODm5xTJfc58lut6uoqEiBgYGyWCzuTkdWq1UxMTE6cuQI+065AJ+36/GZux6fuWvxebsen7nr8Zm7Fp+36xiGoVOnTik6OlpeXi3f1dQlV568vLzUr18/d6fRSFBQEP84XIjP2/X4zF2Pz9y1+Lxdj8/c9fjMXYvP2zVaW3G6gIYRAAAAAOAAiicAAAAAcADFkwfw9/fXvHnz5O/v7+5UugQ+b9fjM3c9PnPX4vN2PT5z1+Mzdy0+b8/UJRtGAAAAAEBbsfIEAAAAAA6geAIAAAAAB1A8AQAAAIADKJ4AAAAAwAEUTwAAAADgAIonF3jhhReUlpam7t27q1evXk2OOXz4sG699Vb16NFDYWFheuyxx1RZWdni61ZUVOjRRx9VWFiYevTooWnTpuno0aNOeAcd26ZNm2SxWJp8bNu2rdl5s2bNajR+7NixLsy8Y4uLi2v0+f3iF79ocY5hGJo/f76io6MVEBCg6667Trt27XJRxh3XwYMH9cADDyg+Pl4BAQEaMGCA5s2b1+r3EM7xtnn11VcVHx+vbt26KSUlRf/4xz9aHL9582alpKSoW7duuuKKK7Ro0SIXZdrxLViwQFdddZUCAwMVHh6u2267TQUFBS3Oae57/Z49e1yUdcc2f/78Rp9dZGRki3M4xy9PU/+ftFgseuSRR5oczznuGXzcnUBXUFlZqbvuukupqalavHhxo+dtNpumTp2qPn366PPPP9eJEyeUkZEhwzD0hz/8odnXfeKJJ/TRRx9p5cqVCg0N1dNPP61bbrlF27dvl7e3tzPfUoeSlpam4uLiBrFf/epX2rBhg0aPHt3i3ClTpmjJkiW1x35+fk7JsbP69a9/rdmzZ9ce9+zZs8XxL730kn73u9/p7bff1uDBg/Wb3/xGN954owoKChQYGOjsdDusPXv2yG636/XXX9fAgQOVl5en2bNn68yZM3r55ZdbnMs57phVq1bpiSee0KuvvqprrrlGr7/+um666Sbl5+erf//+jcYfOHBAN998s2bPnq0VK1boiy++0MMPP6w+ffrozjvvdMM76Fg2b96sRx55RFdddZWqq6v1y1/+UpMnT1Z+fr569OjR4tyCggIFBQXVHvfp08fZ6XYaiYmJ2rBhQ+1xSz9LcI5fvm3btslms9Ue5+Xl6cYbb9Rdd93V4jzOcTcz4DJLliwxgoODG8U//vhjw8vLyygsLKyNvffee4a/v79RXl7e5GudPHnS8PX1NVauXFkbKywsNLy8vIy///3v7Z57Z1JZWWmEh4cbv/71r1scl5GRYfzoRz9yTVKdUGxsrPH73//e4fF2u92IjIw0Fi5cWBs7f/68ERwcbCxatMgJGXZuL730khEfH9/iGM5xx1199dXGnDlzGsQSEhKMX/ziF02Of/bZZ42EhIQGsYceesgYO3as03LszEpLSw1JxubNm5sds3HjRkOSUVZW5rrEOpF58+YZI0aMcHg853j7e/zxx40BAwYYdru9yec5xz0Dl+15gOzsbCUlJSk6Oro2lp6eroqKCm3fvr3JOdu3b1dVVZUmT55cG4uOjlZSUpK2bNni9Jw7sjVr1uj48eOaNWtWq2M3bdqk8PBwDR48WLNnz1ZpaanzE+xEXnzxRYWGhmrkyJF64YUXWryM7MCBAyopKWlwTvv7+2vChAmc05egvLxcvXv3bnUc53jrKisrtX379gbnpiRNnjy52XMzOzu70fj09HR99dVXqqqqclqunVV5ebkkOXROJycnKyoqSpMmTdLGjRudnVqnsnfvXkVHRys+Pl4zZszQ/v37mx3LOd6+KisrtWLFCv30pz+VxWJpcSznuHtRPHmAkpISRURENIiFhITIz89PJSUlzc7x8/NTSEhIg3hERESzc2BavHix0tPTFRMT0+K4m266Se+8844+/fRT/ed//qe2bdum66+/XhUVFS7KtGN7/PHHtXLlSm3cuFFz587VK6+8oocffrjZ8RfO24v/LXBOt913332nP/zhD5ozZ06L4zjHHXP8+HHZbLY2nZtNfV+PiIhQdXW1jh8/7rRcOyPDMPTUU0/p2muvVVJSUrPjoqKi9MYbb2j16tX64IMPNGTIEE2aNEmfffaZC7PtuMaMGaNly5YpKytLb775pkpKSpSWlqYTJ040OZ5zvH395S9/0cmTJ1v8xS7nuGfgnqdLNH/+fGVmZrY4Ztu2ba3eU3NBU79lMAyj1d8+tMecjupS/g6OHj2qrKwsvf/++62+/t13313730lJSRo9erRiY2O1du1a3XHHHZeeeAfWls/8ySefrI1deeWVCgkJ0Y9//OPa1ajmXHz+dqVz+mKXco4XFRVpypQpuuuuu/Tggw+2OJdzvG3aem42Nb6pOFo2d+5cffPNN/r8889bHDdkyBANGTKk9jg1NVVHjhzRyy+/rPHjxzs7zQ7vpptuqv3v4cOHKzU1VQMGDNDSpUv11FNPNTmHc7z9LF68WDfddFODq5AuxjnuGSieLtHcuXM1Y8aMFsfExcU59FqRkZHaunVrg1hZWZmqqqoa/Van/pzKykqVlZU1WH0qLS1VWlqaQ1+3o7uUv4MlS5YoNDRU06ZNa/PXi4qKUmxsrPbu3dvmuZ3F5Zz3F7q47du3r8ni6UJXp5KSEkVFRdXGS0tLm/130Nm19fMuKirSxIkTlZqaqjfeeKPNX49zvGlhYWHy9vZutMrU0rkZGRnZ5HgfH58Wf3mAhh599FGtWbNGn332mfr169fm+WPHjtWKFSuckFnn16NHDw0fPrzZ7wec4+3n0KFD2rBhgz744IM2z+Ucdz2Kp0sUFhamsLCwdnmt1NRUvfDCCyouLq79oXHdunXy9/dXSkpKk3NSUlLk6+ur9evXa/r06ZKk4uJi5eXl6aWXXmqXvDxdW/8ODMPQkiVLdP/998vX17fNX+/EiRM6cuRIgx/su5rLOe9zcnIkqdnPLz4+XpGRkVq/fr2Sk5MlmdeAb968WS+++OKlJdzBteXzLiws1MSJE5WSkqIlS5bIy6vtV2VzjjfNz89PKSkpWr9+vW6//fba+Pr16/WjH/2oyTmpqan66KOPGsTWrVun0aNHX9L3n67GMAw9+uij+vDDD7Vp0ybFx8df0uvk5ORwPl+iiooK7d69W+PGjWvyec7x9rNkyRKFh4dr6tSpbZ7LOe4GbmtV0YUcOnTIyMnJMTIzM42ePXsaOTk5Rk5OjnHq1CnDMAyjurraSEpKMiZNmmTs2LHD2LBhg9GvXz9j7ty5ta9x9OhRY8iQIcbWrVtrY3PmzDH69etnbNiwwdixY4dx/fXXGyNGjDCqq6td/h47gg0bNhiSjPz8/CafHzJkiPHBBx8YhmEYp06dMp5++mljy5YtxoEDB4yNGzcaqampRt++fQ2r1erKtDukLVu2GL/73e+MnJwcY//+/caqVauM6OhoY9q0aQ3G1f/MDcMwFi5caAQHBxsffPCBkZuba9xzzz1GVFQUn3krCgsLjYEDBxrXX3+9cfToUaO4uLj2UR/n+KVbuXKl4evrayxevNjIz883nnjiCaNHjx7GwYMHDcMwjF/84hfGfffdVzt+//79Rvfu3Y0nn3zSyM/PNxYvXmz4+voaf/7zn931FjqUn//850ZwcLCxadOmBufz2bNna8dc/Jn//ve/Nz788EPj22+/NfLy8oxf/OIXhiRj9erV7ngLHc7TTz9tbNq0ydi/f7/x5ZdfGrfccosRGBjIOe5kNpvN6N+/v/Hcc881eo5z3DNRPLlARkaGIanRY+PGjbVjDh06ZEydOtUICAgwevfubcydO9c4f/587fMHDhxoNOfcuXPG3Llzjd69exsBAQHGLbfcYhw+fNiF76xjueeee4y0tLRmn5dkLFmyxDAMwzh79qwxefJko0+fPoavr6/Rv39/IyMjg8/XQdu3bzfGjBljBAcHG926dTOGDBlizJs3zzhz5kyDcfU/c8Mw25XPmzfPiIyMNPz9/Y3x48cbubm5Ls6+41myZEmT32Mu/v0Y5/jl+eMf/2jExsYafn5+xqhRoxq0zc7IyDAmTJjQYPymTZuM5ORkw8/Pz4iLizNee+01F2fccTV3Ptf/fnHxZ/7iiy8aAwYMMLp162aEhIQY1157rbF27VrXJ99B3X333UZUVJTh6+trREdHG3fccYexa9eu2uc5x50jKyvLkGQUFBQ0eo5z3DNZDKPm7j4AAAAAQLNoVQ4AAAAADqB4AgAAAAAHUDwBAAAAgAMongAAAADAARRPAAAAAOAAiicAAAAAcADFEwAAAAA4gOIJAAAAABxA8QQAAAAADqB4AgAAAAAHUDwBAAAAgAP+f0pNswAikUehAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.figure(figsize = (10,4))\n",
    "plt.plot(x,y,\"o-\")\n",
    "plt.plot(x,x,\"o-\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 用两种方式分别实现如下的回归问题的神经网络,注意隐藏层需要使用ReLU激活函数,输出层使用恒等函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"240.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"240.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.4171],\n",
       "        [0.5318],\n",
       "        [0.6598]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=torch.tensor([[1.0,2.0,1.5],[3.0,4.0,2.3],[5.0,6.0,1.2]])\n",
    "\n",
    "h1=torch.nn.Linear(3,3)\n",
    "h2=torch.nn.Linear(3,2)\n",
    "out=torch.nn.Linear(2,1)\n",
    "\n",
    "h1_out=h1(x)\n",
    "h1_out_r=torch.relu(h1_out)\n",
    "h2_out=h2(h1_out_r)\n",
    "h2_out_r=torch.relu(h2_out)\n",
    "out_out=out(h2_out_r)\n",
    "out_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.6419],\n",
       "        [-0.7333],\n",
       "        [-0.6899]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 用pytorch 类的方式把上面的代码改写\n",
    "class Net(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net,self).__init__()\n",
    "        self.h1=torch.nn.Linear(3,3)\n",
    "        self.h2=torch.nn.Linear(3,2)\n",
    "        self.out=torch.nn.Linear(2,1)\n",
    "    def forward(self,x):\n",
    "        h1_out=self.h1(x)\n",
    "        h1_out_r=torch.relu(h1_out)\n",
    "        h2_out=self.h2(h1_out_r)\n",
    "        h2_out_r=torch.relu(h2_out)\n",
    "        out_out=self.out(h2_out_r)\n",
    "        return out_out\n",
    "net=Net()\n",
    "x=torch.tensor([[1.0,2.0,1.5],[3.0,4.0,2.3],[5.0,6.0,1.2]])\n",
    "net(x)\n",
    "#net.forward(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 输出层与隐藏层的激活函数功能不同\n",
    "\n",
    "* 输出层的结果最后要做损失函数进行评价的\n",
    "\n",
    "* 回归与二分类输出层都是一个神经元，但是意义不同\n",
    "\n",
    "* sigmoid函数的作用及双重功能\n",
    "\n",
    "* 代码实现\n",
    "\n",
    "* 多分类的输出层\n",
    "\n",
    "* softmax函数的作用，很少用在隐藏层\n",
    "\n",
    "* dim的意义\n",
    "\n",
    "* 代码实现\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"17.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"17.jpg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.6419],\n",
       "        [0.7707],\n",
       "        [0.8625]], grad_fn=<SigmoidBackward0>)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 用pytorch实现一个二分类神经网络，输入是3个特征，隐藏层有1个神经元，输出层有1个神经元，隐藏层用relu和输出层使用sigmoid激活函数，并提供一个测试样本运行前向传播过程\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(3, 1)\n",
    "        self.fc2 = nn.Linear(1, 1)\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = torch.sigmoid(self.fc2(x))\n",
    "        return x\n",
    "net = Net()\n",
    "x=torch.tensor([[1.0,2.0,0.4],[3.0,4,0.1],[5.0,6.0,0.3]],dtype = torch.float32)\n",
    "net(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输出层的激活函数sigmoid——从回归到二分类\n",
    "\n",
    "\n",
    "\n",
    "Sigmoid函数将实数映射到(0,1)区间,是一个S型的曲线。它的定义形式为:\n",
    "\n",
    "$\\sigma(x) = \\frac{1}{1+e^{-x}}=\\frac{e^x}{e^x+1}$\n",
    "\n",
    "x为输入,$\\sigma(x)$为Sigmoid激活函数的值。\n",
    "\n",
    "Sigmoid函数的主要特点是:\n",
    "\n",
    "取值范围在(0,1)之间\n",
    "单调递增\n",
    "曲线S形,与x轴相交于(0,0.5)\n",
    "导数求解简单\n",
    "存在梯度消失问题\n",
    "Sigmoid函数广泛用于二分类神经网络中,将任意实数映射为0-1之间的概率。但目前由于梯度消失问题,在深层网络中已较少使用。\n",
    "\n",
    "\n",
    "* 在神经网络的输出层加入Sigmoid激活函数,主要有以下目的:\n",
    "\n",
    "1.将网络输出转换为概率\n",
    "Sigmoid函数将实数映射到(0,1)区间,可将网络输出转换为概率值,表示预测的置信度。这对于进行预测很有用。\n",
    "\n",
    "2.实现二分类输出\n",
    "Sigmoid的输出区间为(0,1),可以直接用来表示二分类任务中的概率P(y=1|x)。值接近0表示预测第一类,接近1表示预测第二类。\n",
    "\n",
    "3.定义binary cross entropy loss\n",
    "Sigmoid可与binary cross entropy loss配合使用,构建二分类任务的loss函数,用于训练网络模型。\n",
    "\n",
    "4.提供可微分性\n",
    "Sigmoid函数可导,使得输出层也可导,便于使用梯度下降进行网络参数优化。\n",
    "\n",
    "5.减少梯度消失\n",
    "相比tanh等激活函数,Sigmoid在正负饱和区的梯度更大,一定程度上减轻了梯度消失问题。\n",
    "\n",
    "6.提升收敛性\n",
    "Sigmoid可强制网络输出在一个数值范围内,增强网络的收敛性。\n",
    "\n",
    "总之,Sigmoid适合作为输出层激活函数,将网络输出转换为概率,定义loss函数,实现二分类,并一定程度上提升优化性能。但其梯度消失问题仍限制了网络层数的增长。\n",
    "\n",
    "torch.sigmoid\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=torch.tensor(range(-10,10),dtype = torch.float32)\n",
    "y=torch.sigmoid(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x17b48a76760>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApUAAAEYCAYAAAANlw08AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAxOAAAMTgF/d4wjAAAyb0lEQVR4nO3deXxU9b3/8feZmSwkYcnCEkkmAWJkXwJBRSyKW6WIvaVW21KMgEur3lraq962dtFbirWNtfWnoAWUcvVaxVpbaVXUWqlVEgGh7AGSSTAQskBIQpaZ+f7+SIjETGAgy0kyr+fDPDJzznfm+5nDYXh7vud8j2WMMQIAAADawWF3AQAAAOj5CJUAAABoN0IlAAAA2o1QCQAAgHYjVAIAAKDdCJUAAABoN0IlAAAA2s1ldwEnRUREaODAgXaXAQAAgDYcOXJEdXV1Add1m1A5cOBAFRUV2V0GAAAA2pCUlNTmOoa/AQAA0G6ESgAAALQboRIAAADtRqgEAABAuxEqAQAA0G5Bhcr//M//VGpqqizL0pYtW9pst2LFCp1//vkaMWKEbr31VjU0NHRUnQAAAG0yxignv1wv5hYqJ79cxphe269dn/VMgppS6Mtf/rLuvfdeTZ8+vc02Bw4c0AMPPKBNmzZp8ODBuv766/XUU0/pzjvv7LBiAQBA8Iwxyi2oUH5ptVITojUlJVaWZfW6PosqajR/5UYVltcozOlQg8+v5LgorV4wVUmxUb2qX7s+azAscxbxNjU1Va+88oomTpzYat0jjzyiffv2admyZZKkdevWacmSJdqwYUNQ752UlMQ8lQAAdJBQCTzGGF2R/a4Kymrk838aaZwOS6nxUVq/eEanhFo7+rXrs57qdHmtwyY/93g8SklJaX6empoqj8fTUW8PAACCZIzR/JUbm8NHg88nSSooq9HNKze2O3z4/UZ1Xr/qvL7G3w1+1TZ4dcszuTp07IR8Rs195pdWa+6T7+uHXxglv5H8xsjnb3wPnzHy+U3TslMfK8CyxvZ+f8v1xZUnlF9aLf9nDpH5/Eb7S6v19d99qNjocMlIRkamqQZjJCOp8dDaqc+NjCS/UfOwsjnltScfV55o0IEj1frskTmf32j/kWpd8+g/FBPparH+5GE889kFpyz7tM0p65oeVtd5VVBWE7BPT3mNcgsqlJkaF8SfYOew7Y462dnZys7Obn5eVVVlVykAAHSqrhwS9vuN/r67RIXlLY9mSY3hI7+0Rveu3aqEmAjVNnwaCpsDotevugafapt+159c5vU1tfOr3ucPvh4jHa6s093Pb+ngT3pmxkjv7ytrtdxhSZZlyZJkWZIlS03/ybIkR/O6xt+frrOa2kv1Xn+rcHeqQ5W1iqr9NGad/OM+9U890D5w6qJPX9P4oKrO22Z/YU6H8kure0eodLvd2rdvX/Pz/Px8ud3uNtsvXrxYixcvbn5+utv+AADQU7V3SNgYo+p6n8qq6lRWXa+yqvrmx6VVdY3Pq0/+rld5dX2rMHkqnzF6MTfw8KXTYSnS5VBEmFMRLkfTj1MxkS5FupyKCPt0WYTL0fS8cfm+kmq9u6dEDb7WfYe7HPrKlGRNT4uXw7LkdFhyOCw5Tz5u+u106NP1zctOeWxZcjh0ymNLWwuP6vY1HwXsN8xpac3CC5WZGtcYBjswyOfkl+trT38QsF+X09KKrMwOD3in67PB51dqQnSH9ne2OixUzp07V9OnT9dPfvITDR48WMuWLdNNN93UUW8PAECPc7ph6K//7kM9/tVJnwbFU4LhqQGytKpOdd7THxmMiXApPiZcybF9NDF5gIwxentXSashYUlyOSwtnTtOU1LiPg2FTQHS5Tz3mQZz8sv17p6SNrfD9RPP65SjaJePHKTkuKiA5xm646I0dVhcpxwVnpISe9p+p6TE9oo+z0ZQF+rcfvvteu2113To0CHFx8erb9++ysvL06JFizRnzhzNmTNHkvT0009r6dKlkqTLLrtMy5YtU1hYWFCFcKEOAKC3+fvuEi16Nlfe0xw5DCTc5VBCdLjiYyIUHxOu+OgIJcSEKz4mXHHRjcsSmn7HRYcrMszZ4vWhdhFJoKPB7rgorV54oYYO6NMpfdrVr12f9aTT5bWzuvq7MxEqAQA9md9vtL+0SpsKjmqTp0KbPBXac7jt6wVcDktXjhqkGRcMUnxTgGwMjhGKDne2O4CFWuCxYyoju/q167NKhEoAADpcZW2DtnhOBsij2uKpUGXtpxdSnNc/UikJ0dq4v1y+AP/UhjktPXfrRZ16YUWoBR50vi6ZUggAgN4q0FHIvSVVzVO9hDsdGju0nzLcscpIiVWGO1ZD+keedki4K86BsyxLmalxXXpFsB19onsgVAIAQkawR9FOHoXc3HQkcnOAo5CzxiUqwx2rSe4BGnNeP0W4nK3ex7IsrV4wtc0hYY7goTchVAIAQkJbU/s8mzVVdT7fWR+FDFZSbJTeWjyDIWH0epxTCQDo9doahpYaJ8I+ddF5/SM1qSk8nu4oJBCKOKcSABDScgsqVFR+IuCk4MZIs8clatb4xLM+CgngU4RKAECv9/bOw/L6A08g3ifcqRkXDNSscYldXBXQuxAqAQC91u5Dx/WLv+3SW7sC3+lF6h63twN6A0IlAKDXKT52Qtlv7NHaTUXyG+kL44Zo28FjOni0tlve3g7oDQiVAIBe49iJBj3x9zw988981Xn9mjYiXvdfO1Ljkwac9m4vXIkNtB+hEgDQ49U2+PT7fxXo8XfydOxEg0YO6av7rx2pGekDmwMjU/sAnYtQCQDosfx+o1e2HNSv3tijg0dPaOiAPvrxdaN1/cShcjpah0Xu9gJ0HkIlAKDHMcbo3T1H9PDfdmtncaX69wnTD2aN0jcuTlFkGHNKAnYgVAIAepRtRcf087/u1Pv7yhTucuj2GcP1rRlp6h8VZndpQEgjVAIAeoSCsmr98o09+vPHn8iypBsmJ+k7V6XrvAF97C4NgAiVAIBurqyqTr99O0//+2GBGnxGM0cO0n2fH6kLhvS1uzQApyBUAgC6pZp6r1a8d0DL/7FfVXVeTUgeoP++dqQuGh5vd2kAAiBUAgC6Fa/PrxdyC/Xr9Xt15HidUuOj9PDc8Zo1bgjT/wDdGKESAGALY0yLOSMnuwfojR0l+sXru7T/SLUSYsL10PVjdNNUt8KcDrvLBXAGhEoAQJf77N1t6r1+OR2W6rx+RYU7dc+V52vRpcMVE8E/U0BPwd9WAECXMsZo/sqNKiirkc9v1ODzSZK8fqN+kS6tXzxDg/pF2lwlgLPFeAIAoEvlFlSoqPyEfH7Tat2JBp8KymtsqApAexEqAQBdKr+0Wo42/vUJczqUX1rdtQUB6BCESgBAl4qKcKq2wR9wXYPPr9SE6C6uCEBHIFQCALrMsZoGZb+xR5Lk+MzsQE6HJXdclKakxNpQGYD2IlQCALpEbYNPt67O1b4j1brr8jSlJkQrzGkpKtypMKel1PgorV54IXNRAj0UV38DADqd32/03T98rI355VpwyTB975oL9N2r01vMUzklJZZACfRghEoAQKdbsm6nXttWrFnjhuiHXxglSbIsS5mpccpMjbO5OgAdgeFvAECnWrHhgH634YAyU2OV/ZWJcnz2ZEoAvQKhEgDQaV7bWqz/eW2HRgyM1tPzpygyzGl3SQA6CaESANApNh4o13f+sEUJMRF65papGhAVbndJADoRoRIA0OH2Hj6uRc/mKMxhaVVWppLjouwuCUAn40IdAECHOlxZq6xVOaqp92llVqbGDu1vd0kAugChEgDQYY7XNuiWVTk6ePSEfnnDBH0ufaDdJQHoIgx/AwA6RIPPr2/97ybtKK7Ud69K15cnJ9ldEoAuRKgEALSbMUb3r92m9/aW6qtTk3XXzDS7SwLQxYIOlXv37tW0adOUnp6uzMxMbd++vVUbv9+vxYsXa/To0Ro/frwuv/xy5eXldWjBAIDuJ/vNPVq7qUgzRw7SQ9eP5c44QAgKOlTefvvtuu2227Rnzx7dd999ysrKatXm1Vdf1T//+U99/PHH2rp1q6644gp9//vf78h6AQDdzHMfevTbt/M0Iam/Hv/aJLmcDIIBoSiov/klJSXKzc3VvHnzJElz585VYWFhq6OQlmWprq5OtbW1MsaosrJSSUmcUwMAvdVbOw/rh69skzsuSiuyMhUVzvWfQKgK6m9/YWGhEhMT5XI1NrcsS263Wx6PR2lpn543c9111+mdd97RkCFD1LdvXw0dOlTvvvtu51QOALDVlsKjuuu5zerfJ0zPLpiqhJgIu0sCYKMOHaPIzc3Vv//9bx08eFCffPKJrrjiCt1xxx0B22ZnZyspKan5p6qqqiNLAQB0ooKyai18JkdGRiuyMjUsIdrukgDYLKhQmZycrOLiYnm9XkmNV/l5PB653e4W7VavXq2ZM2dqwIABcjgcuvnmm/XOO+8EfM/FixerqKio+ScmJqadHwUA0BXKqup088qNqqip12+/mqEMd6zdJQHoBoIKlYMGDVJGRobWrFkjSVq7dq2SkpJaDH1L0vDhw/X222+rvr5ekvSXv/xFY8eO7eCSAQB2OVHv08Jnc5VfVqOfXj9WV40ebHdJALqJoM+oXr58ubKysrRkyRL169dPq1atkiQtWrRIc+bM0Zw5c3TnnXdq586dmjBhgsLCwjRkyBAtW7as04oHAHQdn9/o7uc3a0vhUX3rshH6xkUpdpcEoBuxjDHG7iIkKSkpSUVFRXaXAQAIwBijB/70b635wKP/mDRU2V+ZwFyUQAg6XV5jMjEAwBk9+e4+rfnAo0vS4vXw3PEESgCtECoBAKf18qYi/eJvuzVySF89OW+ywl380wGgNb4ZAABt2rC3VPe+tFWJ/SP1zC1T1S8yzO6SAHRThEoAQEA7PqnUHWs+Up9wp55dMFVD+kfaXRKAboxQCQBo5eDRE7rlmY2q9/r11DemKH1wX7tLAtDNcZNWAEALx2oalLVyow5X1uk3X52ki0fE210SgB6AUAkAIc4Yo9yCCuWXVuu8AX302Po92ltSpR/MGqU5E86zuzwAPQShEgBCWFFFjeav3KjC8hqFOR060eCTMdKXM5K06NJhdpcHoAchVAJAiDLGaP7KjSooq5HPb9Tg80mSLEmbCyvsLQ5Aj8OFOgAQonILKlRUfkI+f8sbqxlJnvIa5RYQLAEEj1AJACEqv7RaLmfgO+OEOR3KL63u4ooA9GSESgAIUakJ0Wrw+QOua/D5lZoQ3cUVAejJCJUAEKKmpMRqYN+IVsudDkvuuChNSYm1oSoAPRUX6gBAiDJG6hsRJqlWYU5LYU6HGnx+ueOitHrhhbKswEPjABAIoRIAQtRLm4q0+/BxLZqeqmvGJiq/tFqpCdGakhJLoARw1giVABCCquq8euT13YqLDtfdV6Srf58wZabG2V0WgB6McyoBIAQ98U6ejhyv03evbgyUANBehEoACDGF5TX63YYDGjmkr27KdNtdDoBeglAJACHm53/dqXqvXz+aPVpOB+dOAugYhEoACCEf7C/Tum2HdPXowZqWlmB3OQB6EUIlAIQIn9/owT/vULjToR98YZTd5QDoZQiVABAiXswt1I7iSt0yPVUp8dwtB0DHIlQCQAg4XtugX76xWwkx4brr8jS7ywHQCxEqASAEPP5Onkqr6vW9qy9Q30imEALQ8QiVANDLFZRVa9WGfI1O7KcbpiTbXQ6AXopQCQC93JJ1O1Xv8+tH1zGFEIDOQ6gEgF7s/X2len37YV07doguGh5vdzkAejFCJQD0UqdOIfT9WUwhBKBzESoBoJd6IadQuw4d18JLhyk5LsrucgD0coRKAOiFKmsb9Ks3dmtg3wjdyRRCALoAoRIAeqHfvrVXZdX1+q9rLlBMhMvucgCEAEIlAPQyB0qr9cz7+Ro7tJ++nJFkdzkAQgShEgB6mZ+9tlMNPqMfzR4jB1MIAegihEoA6EU27C3V+p2H9YXxiZo6LM7ucgCEEEIlAPQSXp9fD/5lu8JdDv33tSPtLgdAiCFUAkAv8XxOofYcrtJtlw5XUixTCAHoWkGHyr1792ratGlKT09XZmamtm/fHrDdtm3bdNlll2nUqFEaNWqUXn755Q4rFgAQ2LGaBmW/sVuD+kbom5eNsLscACEo6Hkmbr/9dt12223KysrSSy+9pKysLOXk5LRoU1NTo+uvv16rV6/W9OnT5fP5VF5e3uFFAwBaeuytvaqoadCvbpigaKYQAmCDoI5UlpSUKDc3V/PmzZMkzZ07V4WFhcrLy2vR7rnnntNFF12k6dOnS5KcTqcGDhzYwSUDAE6170iVVv8rXxOS+us/Jg21uxwAISqoUFlYWKjExES5XI3/92tZltxutzweT4t2O3bsUEREhGbPnq2JEydq/vz5OnLkSMdXDQBo9rPXdsrrN/rRdaOZQgiAbTr0Qh2v16v169dr+fLl2rx5s4YOHapvfvObAdtmZ2crKSmp+aeqqqojSwGAkPDuniN6e1eJ5kw4T5NTmEIIgH2CCpXJyckqLi6W1+uVJBlj5PF45Ha7W7Rzu926/PLLNXToUFmWpXnz5umDDz4I+J6LFy9WUVFR809MTEw7PwoAhJYGn18P/WWHIsMcup8phADYLKhQOWjQIGVkZGjNmjWSpLVr1yopKUlpaWkt2n3lK19RTk6OKisrJUnr1q3ThAkTOrhkAIAkPfehR3klVbrtcyN03oA+dpcDIMQFfYng8uXLlZWVpSVLlqhfv35atWqVJGnRokWaM2eO5syZI7fbre9///uaNm2aHA6Hhg4dqqeeeqrTigeAUHW0pl6Prt+jIf0idceM4XaXAwCyjDHG7iIkKSkpSUVFRXaXAQA9wk9e3a5n3s/Xr2+cqC9yxTeALnK6vMYddQCgh8krOa7ff1CgickDNGfCeXaXAwCSCJUA0OM89Jed8vmNfswUQgC6EUIlAPQg7+wq0bt7jug/Jg3VJHes3eUAQDNCJQD0EA0+vx56bYf6hDl17+cvsLscAGiBUAkAPcTv/1Wg/UeqdceMEUrszxRCALoXQiUA9ADl1fX69fo9Oq9/pG77HFMIAeh+CJUA0AM8+uYeVdZ6df+sUeoT7rS7HABohVAJAN3c7kPH9b8fFmhySqyuG59odzkAEBChEgC6MWOM/ue1HfIb6UezR8uymEIIQPdEqASAbuytnSV6b2+p5mYkaULyALvLAYA2ESoBoJuq9/r1s3U7FRXOFEIAuj9CJQB0U6v/la8DpdX61mUjNLhfpN3lAMBpESoBoBsqq6rTY2/t1dABfbToUqYQAtD9uewuAADQyBij3IIK5ZdW640dh3S81qulXxqvyDCmEALQ/REqAaAbKKqo0fyVG1VYXiOnw1Jtg1+RLofGJ/W3uzQACArD3wBgM2OM5q/cqIKyGjX4jGob/JIa7/WdtWqjjDE2VwgAZ0aoBACb5RZUqKj8hHz+luHRZyRPeY1yCypsqgwAgkeoBACb5ZdWy+UMPKl5mNOh/NLqLq4IAM4eoRIAbJaaEK0Gnz/gugafX6kJ0V1cEQCcPUIlANhsSkqszhvQp9Vyp8OSOy5KU1JibagKAM4OoRIAbGZZliY0XeXtdFiKCncqzGkpNT5KqxdeyP2+AfQITCkEADbb/skx/Xlrsaamxup711yggrIapSZEa0pKLIESQI9BqAQAGxlj9OCfd0iSfjxnjMac119Th8XbXBUAnD2GvwHARq9vP6QPD5TrpsxkjTmPic4B9FyESgCwSW2DTz9bt1MxES4tvuoCu8sBgHYhVAKATVb+84AKy0/o7plpGtg3wu5yAKBdCJUAYIOS47X6f2/nKSU+SlmXpNpdDgC0GxfqAIANfvn6blXX+5Q9a5QiXE67ywGAduNIJQB0sX8fPKYXPyrStBHxunr0YLvLAYAOQagEgC50cgohS9KPrhvNPJQAeg1CJQB0oXXbDmljfrm+OtWtkUP62V0OAHQYQiUAdJHaBp+WrNupvpEuLb4q3e5yAKBDESoBoIv87r39Onj0hL59xfmKj2EKIQC9C6ESALrA4cpaPfH3fRqWEK35F6faXQ4AdDhCJQB0gV/8bbdq6n36waxRCnfx1Qug9+GbDQA62ceFR7V2U5EuPT9BV4waZHc5ANApgg6Ve/fu1bRp05Senq7MzExt3769zbbGGM2cOVMDBgzoiBoBoMcyxujBv+yQw5IemM0UQgB6r6BD5e23367bbrtNe/bs0X333aesrKw22z766KMaMWJER9QHAD3an7cW66OCCn39whSlD+5rdzkA0GmCCpUlJSXKzc3VvHnzJElz585VYWGh8vLyWrXdvn27XnnlFd1///0dWykA9DAn6n1aum6n+kW69B2mEALQywUVKgsLC5WYmCiXq/FW4ZZlye12y+PxtGjX0NCgW2+9VcuXL5fTefp72WZnZyspKan5p6qq6hw/AgB0T0+/t1+fHKvVPVemKy463O5yAKBTdeiFOj/96U/1pS99SaNGjTpj28WLF6uoqKj5JyYmpiNLAQBbHTpWqyf/vk/DB0brGxen2F0OAHQ6VzCNkpOTVVxcLK/XK5fLJWOMPB6P3G53i3bvvvuuPB6PHn/8cXm9XlVWVio1NVU5OTkaOHBgp3wAAOiOHv7bLp1o8OmBL4xWmJOJNgD0fkF90w0aNEgZGRlas2aNJGnt2rVKSkpSWlpai3bvvfeeCgoKlJ+frw0bNqhfv37Kz88nUAIIKZs9Ffrj5oOakT5Ql49kCiEAoSHo/31evny5li9frvT0dC1dulSrVq2SJC1atEivvvpqpxUIAD3JySmEnA5LD8w+86lAANBbWMYYY3cRkpSUlKSioiK7ywCAdnll80Hd88IWZU1L1U/mjLG7HADoUKfLa5zoAwAdpKbeq6V/3aUBUWG658rz7S4HALoUoRIAOsjyd/frUGWtvnNlugZEMYUQgNBCqASADvDJ0RNa/o99ShsUo69d6D7zCwCglyFUAkAHePhvu1Tb4NcDs5lCCEBo4psPANrpo4IK/WnLJ5o5cpBmpDOFGoDQRKgEgHbw+40e/PN2uRyWfvAFphACELoIlQDQDn/cfFAfFx3T/ItTNWIgt5sFELoIlQBwjqrrvPrF67sUGxWmb1/BFEIAQhuhEgDO0bJ39+lwZZ0WX32B+keF2V0OANiKUAkA56CookZP/WO/LhjcV1/NTLa7HACwHaESAM7B0r/uUp23cQohF1MIAQChEgDOVk5+uf6ytVhXjhqs6ecn2F0OAHQLhEoAOAuNUwjtUJiTKYQA4FSESgA4C2s3FWnbwWPKmpaqYQnRdpcDAN0GoRIAglRV59UvXt+t+Ohw3c0UQgDQAqESAIL0xDt5OnK8Tt+9+gL1i2QKIQA4FaESAIJQWF6j3204oJFD+upGphACgFYIlQAQhJ//dafqvX796LrRcjosu8sBgG6HUAkAZ/DB/jKt23ZI14wZrGkjmEIIAAIhVALAafj8Rg/9ZYfCnQ59fxZTCAFAW1x2FwAA3ZExRrkFFfpDTqG2f1Kp2z43TCnxTCEEAG0hVALAZxRV1Gj+yo0qLK9Rg89Ikt7YfljzL05VUmyUzdUBQPfE8DcAnMIYo/krN6qg7NNAKUmFFSd088qNMsac5tUAELoIlQBwityCChWVn5DP3zI8+vxGnvIa5RZU2FQZAHRvhEoAOEV+abWcbXwzhjkdyi+t7tqCAKCHIFQCwClio8J1osEfcF2Dz69U7vcNAAERKgGgSU29V795e68k6bPzmzsdltxxUZqSEmtDZQDQ/REqAUCS1+fX3c9t1taiY/rGxW6lJkQrzGkpKtypMKel1PgorV54oSyLu+kAQCBMKQQg5Blj9KNXt+utXSX6UsZQPThnrKTGi3byS6uVmhCtKSmxBEoAOA1CJYCQ98Tf9+m5Dz2anpagpV8a3xweM1PjlJkaZ3N1ANAzMPwNIKSt/ahIj7y+W6MS++nJeRkKd/G1CADngm9PACHrvb1HdN/arTqvf6SeuSVTfSPD7C4JAHosQiWAkLT9k2P65ppNigp36tkFUzW4X6TdJQFAj8Y5lQBCTlFFjW5ZlaN6r1+/XzhV5w/ua3dJANDjESoBhJRjNQ3KWpWjkuN1evxrk3Th8Hi7SwKAXiHo4e+9e/dq2rRpSk9PV2ZmprZv396qzdtvv62pU6dq9OjRGjNmjO699175/YHvTAEAXa22wadbf5+rvJIq/fALozR7/Hl2lwQAvUbQofL222/Xbbfdpj179ui+++5TVlZWqzaxsbH6v//7P+3YsUMfffSR3n//fa1evboj6wWAc+L3G333xY+18UC5FlwyTIsuHW53SQDQqwQVKktKSpSbm6t58+ZJkubOnavCwkLl5eW1aDdp0iQNH974RR0ZGamJEycqPz+/YysGgHOwZN1Ovba1WLPGDdEPvzDK7nIAoNcJKlQWFhYqMTFRLlfjKZiWZcntdsvj8bT5mkOHDumll17S7NmzO6ZSADhHKzYc0O82HFBmaqyyvzJRjs/e2BsA0G6dMqVQZWWlrrvuOt17772aMmVKwDbZ2dlKSkpq/qmqquqMUgCEuHXbivU/r+3QiIHRenr+FEWGOe0uCQB6JcsYY87UqKSkRGlpaSovL5fL5ZIxRomJidqwYYPS0tJatD1+/LiuueYazZo1Sz/84Q+DLiQpKUlFRUVn/wkAoA0bD5Rr3ooP1b9PmF7+5jQlx0XZXRIA9Giny2tBHakcNGiQMjIytGbNGknS2rVrlZSU1CpQVlVV6fOf/7w+//nPn1WgBICOlldyXLeuzlWYw9KqrEwCJQB0sqCHv5cvX67ly5crPT1dS5cu1apVqyRJixYt0quvvipJeuyxx7Rx40a9/PLLmjhxoiZOnKif/exnnVM5ALThcGWtbl6Zo+o6r56cN1ljh/a3uyQA6PWCGv7uCgx/A+gIVXVefWXZv7SjuFKPfHm8bpiSbHdJANBrtHv4GwB6ggafX99c85F2FFfqu1elEygBoAsRKgH0CsYY3b92m97bW6qvTk3WXTPTzvwiAECHIVQC6BWy39yjtZuKNHPkID10/VhZFnNRAkBXIlQC6PGe+9Cj376dp/FJ/fX41ybJ5eSrDQC6Gt+8AHq0t3Ye1g9f2SZ3XJRWZmUqKtxld0kAEJIIlQB6rI8Lj+qu5zarf58wPbtgqhJiIuwuCQBCFqESQI9UUFatBc/kyG+MVmRlalhCtN0lAUBII1QC6HHKqup088qNqqip12+/OkkZ7li7SwKAkMfJRwB6lBP1Pi18Nlf5ZTV66ItjdfWYIXaXBAAQoRJAN2eMUW5BhfJLq+WOi9JT/9ivLYVH9c3LRugbF6XYXR4AoAmhEkC3VVRRo/krN6qwvEZhTodqG3zyG+nq0YP0X1dfYHd5AIBTECoBdEvGGM1fuVEFZTXy+Y0afL7mdXlHqsXc5gDQvXChDoBuKbegQkXlJ+Tzm1brCstrlFtQYUNVAIC2ECoBdEv5pW0fjQxzOpRfWt21BQEATotQCaDb2Vlcqf/90KM6rz/g+gafX6nMSwkA3QrnVALoNg4ePaFfvbFbf9x8UMZIMREunaj3yWc+HQJ3Oiy546I0JYW5KQGgOyFUArDd0Zp6PfH3fXrm/XzVe/2anpag+68dqQFRYS2u/m7w+eWOi9LqhRfK4kodAOhWCJUAbFPb4NMz7+friXfyVFnr1Zjz+un+a0fq0vMHNrd5a/GM5nkqUxOiNSUllkAJAN0QoRJAl/P5jV7eVKTsN/eo+FitkmL76MHrx2rOhPPkcLQMjJZlKTM1TpmpcTZVCwAIBqESQJcxxuid3SV6+K+7tfvwccVGhemB2aM17yK3IlxOu8sDALQDoRJAl9hSeFQ/X7dTHx4oV2SYQ9+6bITuuGyE+kWG2V0aAKADECoBdKoDpdV65PVdWrftkByWdOOUZH3nqnQN6R9pd2kAgA5EqATQKY4cr9Nv3tqr5zd65PUbXTlqsO77/AU6f3Bfu0sDAHQCQiWADlVV59Xv3tuvp/+xX9X1PmW4B+i/Z43iQhsA6OUIlQA6RIPPr//b6NFjb+1VaVW9hg+M1r3XjNQ1YwYzBRAAhABCJYCgGGMCzhdpjNFf/31Ij7y+WwdKqzWwb4R+9h9jdeOUZLmc3AkWAEIFoRLAGRVV1LS6s01yXJS+c2W6frfhgD4uPKrocKcWX5WuRZcOU1Q4Xy0AEGr45gdwWsYYzV+5UQVlNfL5jRp8PknSgSPVuvv5zXI5pKxpqbprZpoSYiJsrhYAYBdCJYDTyi2oUFH5Cfn8psVyI8mypEdvnKjrJgy1pzgAQLdBqATQijFGB0qrtclzVC99VKgGvz9guz5hTtU2BF4HAAgthEoAqq7z6uPCo9rkqdAmz1Ft9lSooqbhjK9r8PmVmhDdBRUCALo7QiUQYk49CrnJU6HNnqPafahSJ0e3w5yWRp/XX1+cNEAZ7lhNcg/Q/BUbVVBe02II3Omw5I6L0pSUWJs+CQCgOyFUAj1QW9P7BHKmo5CD+kbo6tFDlJHSGCLHDu2vyDBni/dYvXBqq6u/3XFRWr3wQuagBABIkixjjDlzs86XlJSkoqIiu8sAur22pvdZvWCqhg7oo/yyGm0qqGgOkYGOQk5KHqCMlFhluAdo6IA+QQXDswmyAIDe6XR5jVAJ9CDGGF2R/W7z9D4nWVbjRTPhTktHT3iblw/qG6EMd+xpj0ICABCs0+U1hr/RK9h1FK0z+jXGqKrOq/LqepVW1ausqk5l1Y2/dxRXKr+0Wp+Z3UfGSDX1PiUOjNYXJyWd9VFIAADaK+hQuXfvXt18880qLS1V//799cwzz2jMmDGt2q1YsUJLly6V3+/XzJkz9cQTTygsLKxDi24PO8JHbwo83bHf0w0HJ8VGdYt+axt8Kq+uV1lVvUqr61TWIizWq+yUZaXV9ar3nv00PVHhTt0xY4RumJLcUR8RAICgBT38PXPmTM2fP19ZWVl66aWX9PDDDysnJ6dFmwMHDuiSSy7Rpk2bNHjwYF1//fW65pprdOedd57x/bti+NuO8NETAk9P7ret4WCnw1JqfJTWL57RrkBrjFGd19/40+BreuzTiXqfbl/zkQ4dq21x1NCypL4RLk0dFtcYIptCY1Wdt+1OJEW4HEqIiVB8TLjio8MV3/Q4Ibrxd1x0uBJiIlRUUaO7n9+sBl/rv7ZhTkvP3XqRMlPjzvnzAgBwOu0+p7KkpERpaWkqLy+Xy+WSMUaJiYnasGGD0tLSmts98sgj2rdvn5YtWyZJWrdunZYsWaINGza0q8iO0Nnho7v02V37dcdF6Y/fnCa/JJ/fyG+MfH7T4nHj78+sN0Z+/6mP1WLZ7sPH9eibe+T97HiwJKdl6cuTkxQfE94cBusa/M2Paxualnn9TcsbH5+6/FyOGEqSw1JTSIxoConhim8KiAkx4YqLbhkao8KdQV8sY8efLQAAUgecU1lYWKjExES5XI3NLcuS2+2Wx+NpESo9Ho9SUlKan6empsrj8bSn9g7T1q3mfH6j/UeqNeORdxQVfubNcboIbtRy5Yl6n4oqTuizL/H5jfYdqdb0h99RZJij6bUt3qjVspPZv+Wy1v0aI9V5fTpyvL5VfSf7nfTgmwpzOZpeb2RM4/v6TdNj0/SOTctPPjemqU3zuk9fa4xpdZ7fqf0eKK3WxIfeDNygk/iM0Qu5hQHXWZYU6XIqMsyhCJdTEWEORbgcio0KV4TLocgwpyJcjqblzpbLXA7tLanS+p2HAx4x7BPm1E/njNFXMjt+GNqyLK1ewPQ+AIDux7YLdbKzs5Wdnd38vKqqqlP7yy+tlstpqd4XeH1NvU9OR2PAO+M/y6dpcOqqWm8bnakx1Hh9fjkjPr0S12p6daBccDIsWM3P1artyddX1NTLYSlgyHNYUv+oMA2MiZBlNb3GalxuyWpcdsrjk31bTa89+bhxXVP7pucHK05o56HjrYK71HgkbXJKrNIGxchpWXI6LDksS06H5HBYn1n2mfWtljW2dzgsFZRV6Ym/7w/Yp8th6edfGqfJKbGKCHMq0uVQRFMwdDmsdgWwnPxyrd95OOA6r9+vYQM77y4zSbFRemvxDKb3AQB0K0GFyuTkZBUXF8vr9TYPf3s8Hrnd7hbt3G639u3b1/w8Pz+/VZuTFi9erMWLFzc/T0pKOpf6g5aaEK0GX+ChTJfT0pPzJnf4uWg5+eX62tMfBDya5XJYevzrGZ1y/tvJfv0B+nU6LP3yhgmd2m+gKO2wpP+65oIO79cYo9e2HQo4HJwSH6UvT07qlLA1JSVWyXFRbQ71d/ZdZizLUmZqHOdPAgC6DUcwjQYNGqSMjAytWbNGkrR27VolJSW1GPqWpLlz5+rVV1/VoUOHZIzRsmXLdNNNN3V81efgZAhwOloGjM4MAXb0GWr9nhwOTomPUpjTUlS4U2HOxvMLO3M42K5+AQDoroK++nv37t3KyspSWVmZ+vXrp1WrVmncuHFatGiR5syZozlz5kiSnn76aS1dulSSdNlll2nZsmVBTSlk19XfJ89FGzqgT6/pMxT7DZXpkwAAsBN31DkF81T23n4BAEDnIlQCAACg3U6X14I6pxIAAAA4HUIlAAAA2o1QCQAAgHYjVAIAAKDdCJUAAABot25z9XdERIQGDhzYZf1VVVUpJiamy/rrqdhOwWE7BYftFBy2U/DYVsFhOwWH7XRmR44cUV1dXcB13SZUdjWmMAoO2yk4bKfgsJ2Cw3YKHtsqOGyn4LCd2ofhbwAAALQboRIAAADtFrKhcvHixXaX0COwnYLDdgoO2yk4bKfgsa2Cw3YKDtupfUL2nEoAAAB0nJA9UgkAAICOQ6gEAABAu/XaUPnaa69p8uTJioiI0D333NNind/v1913360RI0YoLS1Njz/+eJvvs3fvXk2bNk3p6enKzMzU9u3bO7ly+9x5552aOHFi809kZKR+85vfBGx72WWXadiwYc1tH3300S6u1l4/+clPNHDgwObP//Wvf73NtqG0D33Wb37zG40dO1bjxo3T+PHjtWbNmjbbhto+Fex+sWLFCp1//vkaMWKEbr31VjU0NHRxpfaqra3VF7/4RaWnp2vChAm66qqrlJeX16pdfn6+nE5ni++wffv22VCxfVJTU3XBBRc0f/4XXnghYLtQ3qfKyspa7CPp6elyuVwqLy9v0Y796RyZXmr37t1my5Yt5gc/+IH59re/3WLds88+a2bOnGm8Xq8pKyszbrfb/Pvf/w74PpdffrlZtWqVMcaYF1980UyZMqWTK+8eiouLTWRkpCkuLg64fsaMGeaPf/xj1xbVjfz4xz9utV+1JVT3IWOMWb9+vTl69KgxxhiPx2Pi4+NNXl5ewLahtk8Fs1/s37/fJCYmmuLiYuP3+811111nHn/88S6u1F4nTpwwr732mvH7/cYYY37729+aGTNmtGp34MAB079//64trptJSUkxmzdvPm0b9qmWHnnkETN79uxWy9mfzk2vPVJ58v9qXS5Xq3UvvPCCbr31VjmdTsXFxenGG2/U888/36pdSUmJcnNzNW/ePEnS3LlzVVhYGPD/knubZ599Vtdcc42GDBlidyk9WijvQ5J0xRVXqH///pKk5ORkDRkyRIWFhTZXZb9g94uXXnpJc+bM0ZAhQ2RZlu64446A31W9WWRkpGbNmiXLsiRJF110kfLz8+0tqgdjn2ppxYoVWrhwod1l9Bq9NlSejsfjUUpKSvPz1NRUeTyeVu0KCwuVmJjYHEwty5Lb7Q7YtrdZuXLlGf+i3X///Ro3bpxuvPFG7d+/v4sq6z5efPFFTZgwQTNnztQ777wTsE0o70OftX79elVUVCgzM7PNNqGyTwW7XwT7XRVKHnvsMV1//fUB11VXVyszM1MZGRl68MEH5fP5urg6+82fP1/jxo3TwoULdeTIkVbr2ac+9f7776uiokKzZ88OuJ796ez12FB58cUXKyEhIeAPR0ICC3abvffeezp+/LhmzZrV5nv9/ve/165du7R161Zdeumlbf6l7KnOtK3uuOMO5efn6+OPP9ZDDz2kG2+8UQUFBXaX3eWC3ae2bdumW265RS+88IKio6MDvldv36fQfkuWLFFeXp5+/vOft1qXmJiogwcPKicnR+vXr9d7772nX/3qVzZUaZ9//OMf2rp1qzZt2qSEhATdfPPNdpfUra1YsULz588POKLJ/nRuWm/JHuJf//rXOb/W7XaroKBAF198saTGE3LdbnerdsnJySouLpbX65XL5ZIxRh6PJ2DbniDYbbZixQrdfPPNcjqdbbZJTk6W1HiE5a677tL3vvc9lZWVKT4+vkNqtdvZ7F+XXHKJJk2apNzc3BZHAKTetw99VjDbaceOHZo9e7ZWrlyp6dOnt9mut+9Tpwp2v3C73S0uDmjruyoU/PKXv9TLL7+s9evXKyoqqtX6iIgIDRo0SJIUFxenBQsW6LnnntO9997b1aXa5uS+ERYWpnvuuUfp6ekB27BPSVVVVfrDH/6gnJycgOvZn85Njz1S2R433HCDnn76afl8PpWXl+uFF17QjTfe2KrdoEGDlJGR0XzF6tq1a5WUlKS0tLSuLrnLVFZW6qWXXtKCBQvabOP1enX48OHm52vXrtXgwYN75T/+bSkqKmp+vHfvXm3ZskXjxo1r1S4U96FT7dy5U7NmzdJTTz2lq666qs12obZPBbtfzJ07V6+++qoOHTokY4yWLVumm266yY6SbZWdna3nn39eb775pgYMGBCwTUlJSfNVzHV1dXr55Zc1adKkLqzSXtXV1Tp69Gjz8+effz7g52efavTCCy9owoQJGjlyZMD1ob4/nTNbLxPqROvXrzdDhw41ffv2NTExMWbo0KHmT3/6kzHGGK/Xa771rW+ZYcOGmeHDh5tf//rXza/705/+ZBYuXNj8fNeuXeaiiy4y559/vpk8ebLZunVrl3+WrrR8+XLzuc99rtXynJwcc+211xpjjKmqqjKTJ082Y8eONePHjzczZ840W7Zs6epSbTV//nwzZswYM2HCBJORkWFefPHF5nWhvg+d6sorrzQDBgwwEyZMaP7529/+Zoxhn2prv1i4cGHzd5Uxxjz11FNm+PDhZvjw4WbBggWmvr7erpJtUVhYaCSZ4cOHN+9DU6dONcYY88ADD5gnn3zSGGPM2rVrzZgxY8z48ePN6NGjzV133WVqa2vtLL1L7du3z0ycONGMGzfOjB071syZM8ccOHDAGMM+FcjFF19sVq5c2WIZ+1P7cZtGAAAAtFtIDn8DAACgYxEqAQAA0G6ESgAAALQboRIAAADtRqgEAABAuxEqAQAA0G6ESgAAALQboRIAAADtRqgEAABAu/1/UgzwUuKTOEIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x320 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize = (10,4),dpi = 80)\n",
    "plt.plot(x,y,\"o-\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 上图二分类神经网络的定义与前向过程的代码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.5513],\n",
       "        [0.5513],\n",
       "        [0.5513]], grad_fn=<SigmoidBackward0>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 帮我创建一个有一个隐藏层一个输出层的神经网络，输入是3个特征，隐藏层有1个神经元，输出层有1个神经元，隐藏层用relu和输出层使用sigmoid激活函数\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(3, 1)\n",
    "        self.fc2 = nn.Linear(1, 1)\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = torch.sigmoid(self.fc2(x))\n",
    "        return x\n",
    "net = Net()\n",
    "x=torch.tensor([[1.0,2.0,0.4],[3.0,4,0.1],[5.0,6.0,0.3]],dtype = torch.float32)\n",
    "net(x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"16.png\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url= \"16.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.1732, 0.4449, 0.3819],\n",
       "        [0.1798, 0.5519, 0.2683],\n",
       "        [0.1868, 0.6150, 0.1982]], grad_fn=<SoftmaxBackward0>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m在当前单元格或上一个单元格中执行代码时 Kernel 崩溃。\n",
      "\u001b[1;31m请查看单元格中的代码，以确定故障的可能原因。\n",
      "\u001b[1;31m单击<a href='https://aka.ms/vscodeJupyterKernelCrash'>此处</a>了解详细信息。\n",
      "\u001b[1;31m有关更多详细信息，请查看 Jupyter <a href='command:jupyter.viewOutput'>log</a>。"
     ]
    }
   ],
   "source": [
    "# 用pytorch实现一个多分类神经网络，输入是3个特征，第一隐藏层有3个神经元，第二隐藏层有2个神经元，输出层有3个神经元，隐藏层用relu和输出层使用softmax激活函数，并提供一个测试样本运行前向传播过程\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(3, 3)\n",
    "        self.fc2 = nn.Linear(3, 2)\n",
    "        self.fc3 = nn.Linear(2, 3)\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = F.softmax(self.fc3(x), dim=1)\n",
    "        return x\n",
    "net = Net()\n",
    "x=torch.tensor([[1.0,2.0,0.4],[3.0,4,0.1],[5.0,6.0,0.3]],dtype = torch.float32)\n",
    "net(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 多神经元与输出层的softmax函数-从回归到多分类\n",
    "\n",
    "#### 多个神经元的表示\n",
    "\n",
    "多神经元结构允许表示更加复杂的映射关系。每个神经元学习一部分局部特征,最终组合起来增强了整体的表达能力。\n",
    "\n",
    "神经元数量从1增加到n\n",
    "每个神经元有自己的一组参数$w,b$\n",
    "不同神经元可以学习到不同的特征映射\n",
    "最终输出是n个神经元输出的组合\n",
    "这种结构提升了模型拟合复杂问题的能力,是构建神经网络的基础。\n",
    "\n",
    "\n",
    "### 输出层的softmax函数\n",
    "\n",
    "对于一个向量x = [x1, x2, ..., xn],其softmax函数定义为:\n",
    "\n",
    "$$ softmax(x_i) = \\frac{e^{x_i}}{\\sum_{j=n} e^{x_j}} $$\n",
    "\n",
    "即对每个元素xi:\n",
    "\n",
    "$$ softmax(x_i) = \\frac{e^{x_i}}{e^{x_1} + e^{x_2} + ... + e^{x_n}}$$\n",
    "\n",
    "softmax函数的主要性质是:\n",
    "\n",
    "将任意实数向量归一化为概率分布,范围在[0, 1]\n",
    "概率和为1,即$\\sum_{i=1}^n softmax(x_i) = 1$\n",
    "保留了输入元素相对大小的关系\n",
    "\n",
    "softmax函数通常被用在神经网络的输出层,具有以下目的和意义:\n",
    "\n",
    "    1.将网络输出转换为概率分布,表示对各类别的预测概率。\n",
    "\n",
    "    2.实现多分类任务,将类别预测问题建模为概率估计问题。\n",
    "\n",
    "    3.定义多分类交叉熵损失函数,用于模型训练。\n",
    "\n",
    "    4.提供预测结果,选择概率最大的类别作为预测输出。\n",
    "\n",
    "    5.增加模型确定性,生成值域范围固定的概率输出。\n",
    "\n",
    "    6.提供类别之间可比性,概率表示预测的置信度。\n",
    "\n",
    "综上,softmax可将网络输出转换为概率,实现多分类,定义损失函数,最终用于类别预测,这是它被广泛应用于神经网络输出层的主要原因。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### softmax函数的使用\n",
    "\n",
    "在 PyTorch 中，`softmax` 函数用于将输入张量的每个元素转换为概率分布。`dim` 参数指定了在哪个维度上进行 softmax 操作。\n",
    "\n",
    "例如，假设我们有一个形状为 `(3, 2)` 的张量 `x`：\n",
    "\n",
    "```python\n",
    "x = torch.tensor([[1.0, 2.0],\n",
    "                  [3.0, 4.0],\n",
    "                  [5.0, 6.0]])\n",
    "```\n",
    "\n",
    "如果我们调用 `torch.softmax(x, dim=0)`，那么 softmax 操作将在每一列上进行，即：\n",
    "\n",
    "```python\n",
    "torch.softmax(x, dim=0) = tensor([[0.0900, 0.2447],\n",
    "                                 [0.2447, 0.6150],\n",
    "                                 [0.6652, 0.8103]])\n",
    "```\n",
    "\n",
    "如果我们调用 `torch.softmax(x, dim=1)`，那么 softmax 操作将在每一行上进行，即：\n",
    "\n",
    "```python\n",
    "torch.softmax(x, dim=1) = tensor([[0.2689, 0.7311],\n",
    "                                 [0.2689, 0.7311],\n",
    "                                 [0.2689, 0.7311]])\n",
    "```\n",
    "\n",
    "因此，`dim` 参数的作用是指定在哪个维度上进行 softmax 操作，这对于处理多维数据非常重要。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0900, 0.2447, 0.6652],\n",
       "        [0.9362, 0.0171, 0.0466]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([[1, 2, 3], [9, 5, 6]], dtype=torch.float32)\n",
    "torch.softmax(x, dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.2214, 0.5022, 0.2764],\n",
       "        [0.1898, 0.5145, 0.2957],\n",
       "        [0.1608, 0.5241, 0.3151]], grad_fn=<SoftmaxBackward0>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 帮我实现一个多分类神经网络，输入层有三个神经元，第一隐藏层有三个神经元，第二隐藏层有两个神经元，输出层有三个神经元，隐藏层使用relu激活函数，输出层使用softmax激活函数\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(3, 3)\n",
    "        self.fc2 = nn.Linear(3, 2)\n",
    "        self.fc3 = nn.Linear(2, 3)\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = torch.softmax(self.fc3(x), dim=1)\n",
    "        return x\n",
    "net = Net()\n",
    "x=torch.tensor([[1.0,2.0,0.4],[3.0,4,0.1],[5.0,6.0,0.3]],dtype = torch.float32)\n",
    "net(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
